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Abstract 

The  project  manager  is  responsible  for  the  cost, 
schedule  and  perforaance  of  assigned  projects.  In  particu¬ 
lar,  the  cost  of  a  program  is  under  constant  scrutiny  from 
the  Initiation  of  the  program  to  completion.  To  ascertain 
the  cost  effectiveness  of  the  program,  a  cost  estimate  must 
be  derived  that  will  act  as  the  cost  baseline  throughout  the 
program.  The  initial  cost  estimate  is  extremely  important 
because  the  estimate  is  used  to  determine  the  appropriation 
of  funds  for  project  completion.  Inaccurate  cost  estimates 
are  detrimental  to  the  project  and  the  project  manager's 
capability  to  manage  the  project  effectively.  The  paper  will 
describe  an  innovative  approach  to  cost  estimating  that 
Increases  the  overall  accuracy  of  a  cost  estimate  while 
requiring  no  more  manpower  than  previous  methods. 

Program  managers  in  the  Reconnaissance/Electronic  Warfare 
Program  Office  at  Wright-Patterson  Air  Force  Base,  Ohio  are 
responsible  for  the  development  of  electronic  warfare  equip¬ 
ment  that  will  protect  and  enhance  the  effectiveness  of 
weapon  systems  in  future  conflicts.  Electronic  warfare 
systems  are  critical  to  mission  success  in  the  battle  area. 
With  expanding  government  budget  deficits,  program  managers 
of  electronic  warfare  systems  are  tasked  with  the  responsibi- 


lit y  of  procurement  of  effective  and  efficient  systems  that 
must  operate  for  several  years  after  they  are  conceptualized. 

Total  ayatea  coats  aust  be  estiaated  well  in  advance  of 
prograa  Initiation  and  prior  to  Congressional  appropriations 
approval.  Presently,  no  satisfactory  estimating  model  has 
bean  developed  to  eatiaate  these  flight  test  costs.  Without 
the  benefit  of  accurate  estimates  for  fight  test  costs 
goveraaent  funds  aay  be  inappropriately  budgeted;  the  result 
aay  be  lnveataent  in  systems  that  are  less  cost-effective 
than  a  competing  system  that  could  accomplish  the  same  objec¬ 
tives.  To  facilitate  project  decisions,  project  manager's 
auat  be  provided  with  accurate  flight  test  cost  estimate 
The  problea  of  cost  estimating  flight  test  costs  s 
explored.  The  procedures  of  data  collection,  data  normally 
tlon,  aodel  selection  and  aodel  development  were  used  to  find 
a  coat  eatlaating  aodel  for  electronic  warfare  equipment 
flight  teat  coats. 

The  data  collected  consisted  of  historical  flight  test 
coats  and  system  characteristics  (weight,  volume,  type 
ayatea,  etc.)  for  previously  developed  and  fielded  electronic 
warfare  systems.  Next,  the  data  was  normalized  (Inflation 
was  accounted  for).  The  data  was  examined  for  appropriate 
aodel  selection.  Both  parametric  and  nonparametric  techni¬ 
ques  were  explored.  The  techniques  considered  appropriate 
were  linear  regression  and  principal  component  analysis. 


COST  ESTIMATING  MODELS  POE  ELECTRONIC  WARFARE 
EQUIPMK1T  FLIGHT  TESTS 


I.  Introduction 


Accurate  and  timely  cost  estimates  are  an  integral  part 
of  all  acquisition  processes  (15:1).  A  cost  estimate  is  the 
"expected  dollar  cost  to  perform  a  stipulated  task  or  to 
acquire  an  item"  (15:4).  In  particular,  a  requirement 
existed  for  the  Reconnaissance/Electronic  Warfare  Program 
Office  at  Wright-Patter son  Air  Force  Base  (AFB)  to  estimate 
flight  test  costs  for  electronic  warfare  equipment  (2). 
Thus,  the  flight  test  costs  (costs  incurred  to  insure  system 
performance  and  reliability  when  placed  on  an  air  vehicle 
[13])  were  to  be  estimated  for  electronic  warfare  equipment. 
Electronic  warfare  equipment  is  military  equipment  used  to 
exploit,  reduce,  prevent,  or  intercept  the  hostile  use  of  the 
electromagnetic  spectrum  (21:4).  The  electromagnetic  spec¬ 
trum  may  be  considered  as  a  continuum  from  extremely  short 
wave  lengths  (.001  angstroms)  to  extremely  long  wave  lengths 
(1,000,000  kilometers)  which  Includes  visible  light,  heat,  as 
well  as  radar  and  radio  frequencies  (25:460). 

To  meet  budget  requirements,  the  flight  test  costs  must 
be  estimated  up  to  six  years  in  advance  of  the  actual  testing 
(2).  As  a  consequence,  accurate  cost  estimates  were  required 
to  insure  appropriate  decisions  could  be  made  in  the  alloca¬ 
tion  of  funds  for  future  projects.  There  was  no  formally 


recognised  and  endorsed  procedure  to  estimate  the  flight  test 
costs  for  electronic  warfare  equipment.  Therefore,  the  pos¬ 
sibility  existed  that  future  defense  dollars  might  have  been 
inapproprlatly  budgeted,  causing  the  waste  of  government 
funds  or  lack  of  vital  testing. 

Problem  Statement 

The  problem  considered  was  to  develop  a  cost  estimating 
model  capable  of  estimating  electronic  warfare  equipment 
flight  test  costs  up  to  six  years  prior  to  actual  testing. 
Several  reasons  were  identified  as  causes  of  the  problem. 
First,  a  six-year  estimate  required  the  cost  estimates  to  be 
developed  for  systems  that  were  not  fully  designed  or  in  some 
cases  not  even  fully  defined.  Usually,  only  general  know¬ 
ledge  was  known  about  the  system  to  be  developed  (for  exam¬ 
ple,  perhaps  only  the  enemy  system  to  be  countered  would  be 
known — the  enemy  dictated  the  requirement  [30:119]).  In 
addition,  occasionally  only  the  general  type  of  equipment  was 
known.  Examples  of  the  types  of  equipment  were  active  elec¬ 
tronic  countermeasures  (radiating  devices),  passive  measures 
(nonradiating  devices  such  as  radar  warning  receivers),  and 
electromechanical  devices  (chaff  and  flare  dispensers) 
(18:7).  Thus,  the  estimating  requirement  was  complicated  by 
the  need  to  estimate  for  a  wide  range  of  equipment  from 
simple  electromechanical  devices  to  highly  complex  active 
jamming  and  passive  receiving  systems.  Finally,  the  elec¬ 
tronic  warfare  field  was  characterized  by  rapid  technological 
advancements  (18:9,17:1).  These  rapid  technological  advances 


aade  the  coat  eatiaating  process  very  difficult  (2,41). 
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Literature  Review 

The  explosion  of  electronic  technological  advance  over 
the  past  three  decades  has  had  a  profound  lapact  on  the 
developaent  of  electronic  warfare  equipaent  in  the  United 
States  Air  Force  (USAF).  The  lead  over  the  Soviet  Union  in 
electronic  warfare  equipaent  allows  the  USAF  "to  aaneuver  .  . 
Halted  forces  to  critical  points  on  the  battlefield.  With¬ 
out  thea  [electronic  warfare  equipment]  we  would  not  be  able 
to  prevail  against  the  nuaerlcal  imbalance  we  face"  (12:130). 
The  increasing  electronic  threat  and  the  corresponding  vital 
role  of  electronic  warfare  equipaent  has  (and  will)  lead  to 
increased  expenditures  on  electronic  warfare  equipaent.  In 
fiscal  year  (FT)  1981,  $332.7  million  (36. 3Z)  of  the  $884.5 
Billion  Research  Development,  Test  and  Evaluation  budget  was 
spent  on  Test  and  Evaluation  (12:138). 

The  increased  technological  complexity  of  the  threat 
required  a  similar  increase  in  the  technological  complexity 
of  future  electronic  warfare  equipment.  With  the  increased 
advances  in  technology,  aore  flight  testing  will  be  required. 
"Test  and  Evaluation  is  a  critical  step  in  the  aquisition 
process.  Well  planned  and  executed  testing  can  detect  and 
evaluate  problems  before  they  appear  as  deficiencies  in 
deployed  systems  (12:139)." 

The  Increased  defense  dollars  to  be  spent  on  electronic 
warfare  systeas  dictated  a  need  for  developaent  of  a  valid 
aethod  to  predict  electronic  warfare  systea  costs.  As  the 
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total  Research  and  Development  costs  continued  to  Increase, 
better  and  longer  range  Research  and  Development  cost 
estimates  were  required  to  evaluate  Individual  weapon  system 
proposals  and  to  estimate  the  funding  requirements  of  projec¬ 
ted  force  structures  (37:1).  Flight  test  costs  are  a  signi¬ 
ficant  part  of  this  estimate  (37:5).  Furthermore,  the 
avionics  (which  includes  electronic  warfare  equipment)  test 
evaluation  flights  are  the  most  expensive  since  they  require 
extensive  range  ground  tracking  support  and  often  use  air¬ 
craft  as  radar  targets  (6:2). 

The  frequent  changes  in  funding  of  specific  weapon 
systems  and  the  numerous  design  and  technological  changes 
required  to  ensure  a  system  can  successfully  counter  the 
improving  enemy  weapon  systems  suggest  that  "the  matter  of 
uncertainty  is  a  very  real  problem  in  cost  analysis  of  future 
military  systems  and  forces"  (40:265).  This  problem  of 
uncertainty  was  even  more  significant  when  the  complexity 
variance  between  electronic  warfare  equipment  was  considered. 

The  result  of  the  uncertainty  in  cost  analysis  of 
military  systems  in  general  and  electronic  warfare  systems  in 
particular  has  been  large  unpredlcted  cost  overruns.  From 
FY-1975  to  FI-1980  the  unexplained  average  total  system  costs 
(coat  overruns)  varied  from  a  low  of  70%  to  a  high  of  78% 
(31:9).  According  to  John  Allen,  Directorate  of  Program 
Control  for  Electronic  Warfare  equipment  in  Aeronautical 
Systems  Division  Air  Force  Systems  command,  the  estimates  for 
flight  test  costs  of  electronic  warfare  equipment  were  often 
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in  error  by  at  least  200X  (2).  These  flight  test  cost 
overruns  were  aajor  contributing  factors  to  cost  overruns  on 
several  electronic  warfare  aystems,  including  the  Low 
Altitude  Night  Terrain  Infrared  Navigation  Systee  project. 

Specific  'macro'  level  reasons  for  the  cost  overruns 
have  been  identified  as  "inflation,  technical  changes,  quan¬ 
tity  decreases,  overoptiaisa  and  'buy-ins'  and  reduced  DOD 
budget"  (10:105).  Flight  test  costs  are  one  of  the  cost 
eleaents  which  have  led  to  cost  overruns.  Reasons  for  the 
flight  test  cost  overruns  are  not  well  substantiated.  One 
reason  given  for  erroneous  flight  test  cost  predictions  was 
that: 

Costs  of  instruaenting  a  test  vehicle  are  largely 
dependent  upon  the  specific  test  objectives  planned 
for  that  particular  vehicle.  Non-basic  packages  of 
instruaents  are  asseabled  to  satisfy  specific 
flight  test  objectives  and  the  package  is  usually 
installed  at  the  test  site.  It  is  the  cost  of  this 
set  of  instruaents  and  its'  installation  that  are 

included  here,  and  that  cost  varies  considerably - 

[37:48]. 

Capt  Philip  Llnke,  an  Electronic  Warfare  Officer  and 
flight  test  aanager  in  the  4950th  Test  Wing  at  Wright- 
Patterson  AFB,  indicated  that  numerous  items  can  lead  to  wide 
cost  variations  among  different  weapon  system  tests  (29). 
Among  the  items  Capt  Linke  listed  as  elements  which  fostered 
large  cost  variations  were  that  the  agency  using  the  test 
range  facilities  (normally  at  Edwards  AFB,  California;  Nellis 
AFB,  Nevada;  or  Eglin  AFB,  Florida)  was  required  to  pay  for 
scheduled  test  range  time  regardless  of  inflight  or  westher 
aborts  that  precluded  the  teat  aircraft  from  meeting  the 


a 
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scheduled  test  time.  Capt  Linke  also  pointed  out  that  the 
test  aircraft  for  an  electronic  warfare  system  aay  range  froa 
a  relatively  Inexpensive  T-38  or  T-39  aircraft  to  a  much  aore 
expensive  NC-135  or  C-141  aircraft. 

Another  reason  for  the  large  variation  in  flight  test 
costs  was  that: 

Frequently,  comparison  aust  be  aade  between  hard¬ 
ware  iteas  which  are  not  similar,  or  coaparable  on 
the  basis  of  one  aeaaureaent.  Relative  coaplexity 
aust  be  Identified.  Several  of  the  factors  which 
aay  aid  in  the  deteraination  of  this  coaplexity 
are:  size,  voluae,  weight,  density,  type  of 
aaterials,  nuaber  of  parts,  and  performance 
paraaeters  [1:30]. 

David  Benoy  (7:2),  John  Allen  (2),  and  Jane  Robbins  (41) 
concur  and  a  review  of  all  available  literature  substantiates 
that  no  accurate  aeaaureaent  tool  has  been  developed  to 
estiaate  flight  test  costs  of  electronic  warfare  equipaent. 
The  previously  aentioned  factors  of  electronic  warfare 
syateas'technologlcal  coaplexity,  electronic  warfare  systems' 
diversity  (sice,  weight,  voluae,  density,  type  of  materials, 
nuaber  of  parta,  and  performance  parameters),  and  the 
variability  of  costa  associated  with  test  range  use  were  all 
indicated  as  factors  which  contributed  to  the  lack  of 


development  of  a  reliable  model  to  estiaate  flight  test  costs 


(2;  10:105;  20;  23;  27;  30;  41). 
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1.  Reimbursable  flight  test  costs:  costs  incurred  by  a 
program  office  as  a  result  of  requested  testing  (29).  In  the 
last  several  years  program  offices  have  absorbed  increasing 
proportions  of  cost  center  expenses;  currently  almost  all 
flight  test  "cost  center  expenses  are  reimbursable  with  the 
exception  of  military  labor  and  normal  base  support  func¬ 
tions"  (7:1).  In  this  thesis  the  term  "flight  test  costs" 
will  refer  to  the  flight  test  center  costs  incurred  in 
support  of  electronic  warfare  equipment  testing  that  can  be 
charged  as  reimbursable  expenses  to  the  program  office 
responsible  for  the  development  of  the  specific  electronic 
warfare  equipment  being  tested. 

2.  Electronic  warfare  equipment:  military  equipment 
used  "to  prevent  or  reduce  an  enemy's  effective  use  of 
radiated  electromagnetic  energy  and  actions  taken  to  insure 
our  own  effective  use  of  radiated  electromagnetic  energy" 
(45:250). 

3.  Electronic  warfare  equipment  categories:  electronic 
warfare  equipment  is  extremely  diversified  and  was  qualita¬ 
tively  divided  in  order  that  cost  factors  could  be  objec¬ 
tively  analyzed.  Four  sub-categories  of  electronic  equipment 
were  defined: 

a.  Receivers:  electronic  warfare  equipment 
designed  to  intercept,  interpret  and  display  electronic 
signals  in  order  that  the  position  and  danger  posed  by 
enemy  radars  may  be  known  (21:3). 

b.  Jammers:  electronic  warfare  equipment  designed 


to  deceive  and  disrupt  enemy  radars  by  noise,  barrage  or 
repeater  signal  scraabling  (21:3). 

c.  Dispensers:  electronic  warfare  equipaent 
designed  to  deceive  enemy  radars  and  missiles  through 
the  use  of  chaff  and  flare  deployment  (21:4). 

4.  Key  flight  test  site  cost  elements: 

a.  Teat  Vehicle  Fabrication:  the  cost  of  flight 
teat  vehicles  during  the  Research  and  Development  period 
(37:9). 

b.  Teat  Operations:  those  activities  associated 
with  flight  testing  vehicles  required  by  the  development 
program  (37:9). 

c.  Test  Ground  Support  Equipment:  that  equipment 
required  at  the  test  sites  for  flight  test  activities 
(37:9). 

d.  Test  Instrumentation:  test  vehicles  designed 
to  collect  data  required  by  the  test  objectives  (37:10). 

e.  Fuels,  Propellants  and  Gases:  Mthe  fuels, 
propellants  and  gases  used  during  flight  test 
operations"  (37:10). 

f.  Data  Reduction  and  Analysis:  "the  activities 
associated  with  the  processing  for  analysis  and  study 
the  measurements  of  performance  recorded  during  develop¬ 
mental,  captive,  and  flight  test  operations"  (37:10). 

g.  Maintenance,  Supply,  Miscellaneous:  "those  acti¬ 
vities  associated  with  maintenance  and  supply  of  test 
vehicles,  test  ground  support  equipaent,  and  installa- 


tions  and  facilities”  (37:10). 

5.  Cost  Batiaating  Relationships:  "An  analytical  expres¬ 


sion  which  describes,  for  predictive  purposes, the  quantity  or 
cost  of  an  itea  or  activity  (either  in  physical  units  or 
dollars)  as  a  function  of  one  or  aore  explanatory  variables" 
(15:4). 

6.  Coefficient  of  Determination  (R2);  A  aeasureaent  of 
the  proportionate  reduction  in  total  variation  in  a  dependent 
variable  that  can  be  explained  by  the  use  of  independent 
variable(a)  (28:97,241). 

7.  Independent  Variable:  The  variables  that  influence  or 
that  can  be  used  to  predict  the  flight  test  costs  of  electro¬ 
nic  warfare  equipment. 

8.  Dependent  Variable:  The  variable  we  are  interested  in 
measuring:  in  this  research  project  the  dependent  variable  is 
the  flight  test  costs  of  electronic  warfare  equipment  that 
are  chargeable  to  the  program  office  responsible  for  the 
development  of  the  electronic  warfare  system. 

9.  Colllnear ity :  The  correlation  among  independent 
variables  (28:272). 

Scone 

A  prediction  model  was  designed  which  used  the  reimburs¬ 
able  coats  that  a  program  office  incurs  in  the  development 
and  testing  of  electronic  warfare  equipment.  The  model  was 
tailored  to  the  needs  of  the  Electronic  Warfare  System 
Program  Office,  Aeronautical  Systems  Division,  Wright- 
Pattereon  APB.  The  model  was  tailored  for  one  or  more  of  the 


electronic  squlpaent  categories  (receivers,  jammers,  dispen¬ 
ser).  The  aodel  eaployed  only  those  input  prediction  factors 
which  could  be  clearly  defined  or  accurately  estimated  as 
early  as  six  years  prior  to  the  date  the  flight  test  costs 
were  to  occur. 

Reaeerch  Questions 

The  following  are  the  geraane  research  questions  formu¬ 
lated  for  the  problea  of  estlaatlng  long  range  flight  test 
costs  for  electronic  warfare  equipaent: 

1.  Does  the  historical  data  available  represent 
the  population  to  be  predicted? 

2.  Are  the  coat  estiaating  (causal)  relationships 
predicted  by  experts  in  the  field  consistent  with  the 
data  collected? 

3.  Whet  type  of  aodel  is  the  aost  appropriate  for 
this  problea? 

A.  Vhich  explanatory  variables  can  be  identified 
to  predict  flight  test  costs? 

5.  How  accurately  can  electronic  warfare  flight 
teat  costs  be  predicted  with  the  aodel  and  variables 
selected? 

6.  Can  the  aodel  be  validated? 

7.  Does  the  aodel  fulfill  the  needs  of  the  Recon¬ 
naissance/Electronic  Warfare  Systeas  Prograa  Office? 

Availability  of  Data 


Data  was  obtained  froa  the  Aeronautical  Systeas  Division 


(ASD)  cost  library;  the  flight  test  ranges  (Edvards  AFB, 
California;  Eglin  AFB,  Florida;  and  Nellis  AFB,  Nevada);  the 
ASD  Syateas  Prograa  Offices  at  Vright-Patterson  AFB,  Ohio; 
the  4950th  Flight  Test  Wing  at  Urlght-Patterson  AFB,  Ohio, 
and  fron  an  unpublished  thesis  by  Thomas  J.  DuPre'. 

The  ASD  cost  library  had  a  nunber  of  docuaents  with 
required  data.  The  most  noteworthy  published  data  base  was 
the  September  1980  Rand  Corporation  document  Special  Duty  and 
Coabat  Avionics  Data  Base(s)  (19).  The  document  contained  an 
extensive  list  of  electronic  warfare  equipment  in  the  USAF 
inventory.  Data  included  100th  unit  cost,  weight,  learning 
curves,  volume,  density,  and  technological  year  for  the 
equlpaent.  Additional  docuaents  from  which  data  were 
obtained  are  listed  in  the  bibliography  (3,  4,  17,  16,  19, 
20,  30). 

A  second  source  of  data  was  the  flight  test  ranges 
(Edvards  AFB,  Eglin  AFB,  and  Nellis  AFB)  where  the  tests  were 
accomplished.  Although  useful  in  Boat  cases,  some  of  the  data 
obtained  from  the  test  ranges  were  fragmented.  The  most 
reliable,  comprehensive  and  useful  data  was  obtained  from  Mr. 
John  Killlngsvorth  of  the  3246th  Test  Wing  at  Eglin  AFB, 
Florida.  Another  noteworthy  source  was  the  Management  Infor¬ 
mation  System  Test  and  Evaluation  (MISTE)  system  at  Edwards 
AFB,  California  (42).  The  third  source  of  data  was  the  indi¬ 
vidual  prograa  offices  of  ASD  at  Vright-Patterson  AFB,  Ohio. 
The  financial  managers  for  each  prograa  office  (one  prograa 
office  for  each  electronic  varfere  system  developed  or  modi- 
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fled)  collected  end  kept  a  detailed  record  of  all  the  flight 
teat  coats  for  a  particular  system.  Unfortunately,  no  set 
procedure  had  been  established  to  retain  the  flight  test  cost 
records  after  an  acquisition/aodif ication  prograa  was 
completed  (2,41).  This  problea  is  addressed  in  the  recommen¬ 
dations  section  of  this  docuaent. 

The  next  source  of  data  was  the  4950th  Test  Wing  at 
Vright-Patterson  AFB,  Ohio.  The  4950th  is  responsible  for 
the  early  developmental  flight  testing  (29).  Data  obtained 
was  Halted  to  four  programs  because  the  unit  also  does  not 
have  an  established  procedure  for  maintaining  historical 
flight  test  costs  (9). 

The  final  data  source  was  the  unpublished  thesis  by  Lt 
Thoaas  J.  DuPre'.  The  data  contained  in  the  document  had 
been  obtained  from  some  of  the  previously  mentioned  sources 
(20). 
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II.  Background 

Flight  teat  costa  are  the  reiabureable  costa  that  are 
charged  to  a  prograa  office  as  the  result  of  a  test  flight 
activity  (7:1).  The  following  will  explain  the  sequence  of 
events  as  well  as  the  types  of  activities  that  cause  flight 
test  costs  to  be  incurred. 

First,  when  a  prograa  office  deteraines  a  requirement 
exists  to  flight  test  their  systeas  (or  a  part  of  the  system) 
a  prograa  introduction  docuaent  (PID)  is  subaitted  to  an  Air 
Force  Flight  Test  Center.  "The  PID  defines  specific  systems 
specifications  and  requireaents  to  be  tested"  (7:1).  Next, 
the  PID  is  sent  through  the  various  offices  at  the  Flight 
Test  Center  and  estiaated  costs  are  attached  for  each  of  the 
requested  tests.  The  final  product  is  then  sent  back  to  the 
originating  prograa  office  through  a  statement  of  capability 
(SOC).  This  docuaent  (SOC)  indicates  to  the  prograa  office 
the  expected  costs  to  coaplete  the  required  testing. 

The  following  are  typical  costs  that  are  Incurred  for 
flight  testing  (13): 

1.  Aircraft  fuel  costs  (POL) 

2.  A  cost  per  flying  hour  for  all  needed  aircraft 

3.  Logistic  support  for  sll  aircraft 

4.  Coaputer  support 

5.  Engineering  support 

6.  Range  support 

7.  Test  requireaents  support 
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8. Aircrew  support  (military  aircrew  costs  are  not 


included) 

9.  Photographies 

10.  Electronic  support 

11.  Miscellaneous  costs 

The  purpose  of  our  model  is  to  predict  the  aforemen¬ 
tioned  costs  (the  total  cost  for  all  "he  flight  tests)  well 
in  advance  of  the  actual  determination  of  what  tests  will  be 
needed  for  an  acquisition/modification  program. 

Analytical  Techniques 

The  difficulty  of  estimating  the  cost  of  electronic 
warfare  equipment  that  "is  characterized  by  rapid  technologi¬ 
cal  change"  was  the  major  factor  in  determining  the  analyti¬ 
cal  techniques  and  procedures  most  appropriate  for  resolving 
the  problem.  A  brief  discussion  of  the  applicability  of  the 
techniques  available  illustrates  the  constraints  that 
prohibited  a  simple  solution  to  the  problem;  a  detailed 
discussion  of  the  specific  procedures  to  be  used  is  found  in 
Chapter  III. 

The  first  requirement  for  any  analytical  technique  and 
specifically  for  a  cost  estimating  relationship  is  that  all 
predicting  relationships  be  based  on  logic  and  causality 
(16:16).  If  a  causal  relationship  is  not  established  between 
the  Independent  variables  that  are  used  to  predict  the  depen¬ 
dent  variable  (cost)  we  are  Interested  in,  the  independent 
variables  cannot  be  relied  on  to  yield  consistent  results 
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over  time  as  other  factors  intervene.  Ideally,  the  technique 
aalectad  would  utilize  predictive  information  from  all 
significant  factors  that  determine  the  flight  test  costs  of 
electronic  warfare  equipment.  Both  nonparametric  and  para¬ 
metric  estimating  techniques  are  currently  used  by  Aeronau¬ 
tical  Systems  Division  at  Vr ight-Patter son  AFB  to  estimate 
electronic  warfare  equipment  costs  (2).  Nonparametric 
approaches  rely  on  estimating  costs  by  evaluating  elementary 
components  of  a  system  and  then  summing  the  expense  of  all 
components  to  arrive  at  a  total  cost  for  the  system  (38). 
The  nonparametric  approach  was  judged  to  be  inappropriate  for 
the  estimation  of  flight  test  costs  of  electronic  warfare 
equipment  since  these  estimates  often  must  be  done  up  to  six 
years  in  advance  to  meet  budgeting  constraints.  Six  years 
prior  to  flight  testing  a  system  is  often  not  fully  concep¬ 
tualized  or  defined  (2)  and  therefore  does  not  allow  compo¬ 
nent  part  costs  to  be  estimated. 

Parametric  approached  involve  the  selection  of  indepen¬ 
dent  variables  which  have  strong  causal  linkages  to  the 
dependent  variable  or  item  of  interest  (38).  In  this  case  we 
were  interested  in  the  cost  estimating  relationships  between 
independent  variables  that  can  be  selected  several  years  in 
advance  of  actual  flight  test  costing  and  the  flight  test 
coata  that  actually  occur.  In  solving  this  problem  we  must 
first  logically  determine  how  certain  key  factors  will  affect 
the  development  of  an  electronic  warfare  system  and  then 
estimate  the  funding  that  will  be  required  to  adequately 
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flight  test  the  new  system.  Selecting  the  independent 
variables  that  accurately  predict  impacts  on  electronic  war¬ 
fare  system  development  has  been  considered  to  be  a  difficult 
task: 

The  choice  of  physical  or  performance 
characteristics  for  estimating  the  cost  of  a  given 
variety  of  electronic  warfare  equipment  is  crucial. 

As  already  pointed  out,  electronic  warfare  equip¬ 
ment  must  be  broken  out  into  a  sizeable  number  of 
homogeneous  classes  before  valid  estimating 
relationships  can  be  prepared.  In  addition,  it  is 
rare  that  a  single  characteristic  serves  as  an 
adequate  measure  of  the  equipment  cost  within  a 
single  class— sultivariate  relationships  are 
generally  required.  It  also  appears  that  the  range 
of  size  and  performance  over  which  a  relationship 
is  valid  is  limited  to  a  considerably  greater 
extent  than  is  true  for  airframes  and  power  plants. 

Thus,  it  is  desirable  in  deriving  electronic  equip¬ 
ment  estimating  relationships  to  test  the  range  of 
values  over  which  these  functions  are  usable  and 
specify  the  limitations  of  each  relationship  [5:5]. 


It  has  been  stated  that  "the  primary  criterion  for 
choosing  parameters  for  estimating  is  that  they  provide  the 
most  sensitive”  measures  of  equipment  design  and  that  they 
"be  readily  available  early  in  the  design”  (5:6).  Interviews 
with  test  experts  indicated  that  weight,  density,  volume, 
power  requirements  and  whether  or  not  the  system  developed 
was  a  modification  of  an  old  system  would  all  impact  on  total 
equipment  costs  and  the  testing  costs  required  to  validate 
that  system.  Additionally,  the  specific  equipment  category 
of  the  system  (receiver,  jammer,  dispenser)  would  all  impact 
upon  flight  test  costs  (2;29). 

A  widely  used  parametric  estimating  method  is  the  linear 
regression  technique.  A  major  limitation  of  this  technique 


Is  that  if  two  or  gore  important  explanatory  variables  are 
collinear  the  regression  analysis  may  be  in  serious  error 
(35:196).  With  collinearity  little  confidence  can  be  expres¬ 
sed  in  the  prediction  coefficients;  collinearity  can  be 
detected  by  high  correlation  coefficients  between  independent 
variables  (35:196).  A  preliminary  statistical  analysis  of 
the  data  indicated  high  correlation  coefficients  between 
several  of  the  important  independent  variables;  this 
indicated  the  existence  of  a  serious  collinearity  problem 
that  would  significantly  limit  the  confidence  that  could  be 
placed  in  any  results  achieved  with  a  linear  regression 
model. 

A  recent  parametric  technique  that  eliminates  the 
effects  of  collinearity  is  the  use  of  principal  component 
analysis  (22:38).  This  regression  technique  computes  the 
value  of  dependent  variables  by  forming  an  equation  made  up 
of  principal  components  that  are  derived  from  various  combi¬ 
nations  of  the  original  independent  variables  (22:38).  These 
principal  components  retain  all  the  statistical  information 
possessed  by  the  original  independent  variables  but  are 
completely  free  of  collinearity.  These  principal  components 
are  then  used  as  'statistically  pure*  independent  variables 
in  an  equation  to  estimate  the  value  of  the  dependent  vari¬ 
able  (flight  test  costs)  (22:36).  Because  of  the  major 
difficulties  of  collinearity  and  the  limited  data  available, 
the  principal  component  analysis  method  was  selected  ss  the 
primary  analytical  technique  to  resolve  this  difficult  prob- 
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lea;  the  resulting  capability  to  retain  all  data  with  infor- 
aational  value  was  essential  since  other  techniques  evaluated 
would  have  required  soae  of  the  data  to  be  discarded.  A  full 
discussion  of  the  principal  component  aethod  is  in  Chapter  3 
(Methodology) . 


Assumptions 

1.  The  less  that  is  known  about  a  system  the  more  it 
will  have  to  be  tested  to  verify  its  capability.  This  will 
noraally  Increase  flight  test  costs. 

2.  The  aore  an  electronic  warfare  system  changes  in 
technology,  capability  or  method  of  utilization  from  previous 
systems  the  more  it  will  cost  to  flight  test  that  system. 
This  assumption  follows  from  assumption  #1. 

3.  Variables  that  explain  the  variability  in  electronic 
warfare  system  costs  will  also  help  explain  the  variability 
in  electronic  warfare  equipment  flight  test  costs.  This 
assumption  follows  from  assumption  #1  and  #2. 

4.  A  qualitative  variable  reflecting  the  mean  year  of 
the  years  that  the  system  was  flight  tested  will  capture  the 
lapact  that  (1)  technological  change  and  (2)  the  increases  in 
the  coats  at  each  test  site  have  on  total  flight  test  costs. 

5.  A  qualitative  variable  reflecting  the  type  of  air¬ 
craft  used  for  flight  testing  will  be  of  value  in  predicting 
flight  test  costs. 

6.  A  qualitative  variable  reflecting  whether  or  not  the 
system  considered  is  a  aodiflcation  to  an  existing  system 


:v::- 


>  .*•  .*•  / 


will  be  of  value  In  predicting  flight  test  costs.  This 
assuaption  follows  from  assumptions  #1  and  #2. 

7.  Factors  that  increase  the  flight  test  costs  of 
equipment  in  each  of  the  four  categories  of  electronic  war¬ 
fare  equipment  (receivers,  jammers,  missiles,  and  dispensers) 
will  also  help  predict  the  increases  in  flight  test  costs  of 
equipment  in  the  other  three  electronic  warfare  equipment 
categories.  This  assumption  follows  from  the  similar  test 
requirement  for  all  categories  of  equipment. 

Model  Use 

The  model  developed  will  be  used  by  cost  analysts  at  the 
Reconnaissance/Electronic  Warfare  Program  Office  at  Wright- 
Patteraon  AFB  to  predict  the  flight  test  costs  for  electronic 
warfare  equipment.  The  cost  estimate  for  flight  tests  is 
only  one  of  eight  cost  predictions  required  for  a  budget 
eatimate.  The  other  areas  that  costs  must  be  estimated  for 
are  (2,20:23): 

1.  Prime  mission  equipment 

2.  Peculiar  support  equipment 

3.  Software 

4.  System  integration 

5.  Data 

6.  Training 

7.  Miasion  support 

At  present,  there  are  adequate  techniques  and  models 
available  for  the  preceding  seven  areas;  however,  no  adequate 


technique  or  aodel  exists  for  predicting  future  flight  test 
costs  for  slsctronic  warfare  equipment  (2).  Therefore,  the 
aodsl  will  be  used  to  predict  long  range  flight  test  costs 
that  will  be  used  for  budget  subaissions. 


Data  Collection 

The  data  collected  is  an  accurate  reflection  of  the 
population  for  which  the  aodel  was  developed  (2).  The  popu¬ 
lation  consists  of  weapons  designed  to  negate  the  electronic 
threat  of  the  enemy  in  order  that  USAP  air  power  can  be 
Maneuvered  to  "initial  points  on  the  battlefield"  (12:130). 
Electronic  warfare  equipment  will  be  crucial  in  determining 
the  outcome  of  future  air  battles. 

This  research  has  placed  electronic  warfare  equipment  in 
three  categories:  receivers,  jammers,  and  dispensers. 
Operational  definitions  of  each  type  of  electronic  warfare 
equipment  are  found  in  Chapter  I.  Therefore,  the  population 
applicable  to  this  research  consists  of  all  past  and  present 
electronic  warfare  equipment  used  on  aircraft  that  can  be 
categorised  as  a  receiver,  jammer,  or  dispenser. 

The  population  under  consideration  evolves  in  response 
to  changing  Air  Force  requirements.  Major  Commands 
(primarily  Strategic  Air  Command;  Tactical  Air  Command; 
United  States  Air  Forces  Europe;  and  Pacific  Air  Forces) 
generate  Statements  of  Operational  Need  that  detail  the 
requirements  current  or  future  systems  must  possess  in  order 
to  successfully  counter  the  threats  posed  by  enemy  capabili¬ 
ties  (11:3-1,3-2).  The  Stateaent  of  Operational  Need  is 
acted  upon  and  the  foundation  for  the  creation  of  a  new 
ayatea  is  completed  early  in  the  conceptual  phase  of  the 
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Defense  Systems  Acquisition  Review  process  (14:60).  At  this 
point  s  progrsa  office  is  normally  assigned  responsibility 
for  the  system.  If  the  system  is  electronic  warfare  equip¬ 
ment  it  is  often  assigned  to  the  Reconnaissance/Electronic 
Warfare  Program  Office  at  Wright-Patterson  AFB;  normally, 
forty  to  fifty  separate  electronic  warfare  system  are  in 
development  under  the  direction  of  the  Reconnaissance/Elec¬ 
tronic  Warfare  Program  Office  (2).  These  systems  comprise 
the  major  portion  of  additions  to  the  population  this 
research  is  concerned  with  although  other  appropriate  members 
of  the  population  come  from  the  Electronic  Systems  Division 
at  Hanscom  Field  in  Boston,  Massachusetts  and  the  United 
States  Department  of  Naval  Operations  electronic  warfare 
projects  (2). 

The  data  collected  is  representative  of  the  population 
of  Interest  (2).  Each  of  the  equipment  categories  is  repre¬ 
sented  in  the  sample  in  approximately  the  same  porportion  as 
it  exists  in  the  population.  Additionally,  our  use  of  dummy 
variables  to  account  for  differences  in  costs  in  each  of  the 
equipment  categories  allowed  some  latitude  in  not  having  the 
sample  percentages  in  each  category  exactly  reflect  the  popu¬ 
lation  percentages  (see  Assumptions  #7  Chapter  II).  Dummy 
variables  are  independent  variables  used  to  indicate  cate¬ 
gorical  differences  between  observations.  If  enough  data 
were  available  an  optimum  predictive  model  could  be  built  for 
each  electronic  warfare  category. 

Details  of  the  data  collection  effort  are  found  in 


Chapter  I,  Availability  of  Data.  The  data  was  normalized 
through  the  uae  of  inflation  factors  (all  cost  figures  are  in 
FT  82  dollars)  and  a  variable  that  accounted  for  advance  in 
the  technological  complexity  of  electronic  warfare  equipment 
(16:26).  A  variable  was  also  used  to  Indicate  whether  or  not 
the  electronic  warfare  system  was  a  new  or  modified  system; 
this  measured  the  "learning  curve"  phenomenon  that  was 
asaumad  to  exist  (16:26).  Additionally »  the  data  were 
analyzed  and  adjusted  to  ensure  the  measurement  indices  from 
the  different  flight  test  centers  were  consistent  (16:26). 

In  the  data  collection  process  test  engineers  were  ques¬ 
tioned  aa  to  what  they  thought  the  most  important  variables 
would  be  in  predicting  flight  test  costs.  They  were  also 
aaked  to  indicate  how  these  independent  variables  would 
impact  the  flight  test  costs.  The  responses  of  the  test 
engineers  were  instrumental  in  constructing  the  model  (see 
Developing  the  Model  in  this  chapter). 

Model  Selection 

The  main  criteria  for  model  selection  were  to  determine 
whether  or  not  the  model  could  sccurately  estimate  the  cost 
of  flight  testing  specified  electronic  warfare  equipment  and 
to  ensure  that  the  selected  model  could  be  used  by  the  Recon¬ 
naissance/Electronic  Warfare  Program  Office  personnel  to 
predict  coats  up  to  six  years  in  advance  of  the  flight  tests. 

Nonparametrlc  techniques  rely  on  a  building  block 
approach  to  cost  estimating.  The  cost  of  individual  subcom- 
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ponents  and  components  are  estimated  and  then  auaaed  to 
calculate  an  overall  coat  for  the  system  (38).  Thla  techni¬ 
que  cannot  be  uaed  to  eatlaate  the  coata  of  flight  teating 
electronic  warfare  equipaent  up  to  aix  jeara  prior  to  the 
flight  testing  because  at  the  six  year  point  in  the  develop- 
aent  cycle  the  system  is  often  not  fully  conceptualized  or 
defined  (2).  The  building  blocks  of  the  flight  test  costs 
cannot  be  calculated  until  the  system  is  conceptualized  and 
designed.  Therefore,  all  nonparametric  cost  estimating 
techniques  were  eliminated  because  the  necessary  data  would 
notbe  available  to  the  cost  analysts  of  the  Reconnaissance/ 
Electronic  Warfare  Program  Office  at  the  time  flight  test 
costs  are  estimated. 

Parametric  estimating  techniques  were  considered  pri¬ 
marily  because  indications  were  that  the  parametric  methods 
would  fulfill  the  criteria  of  being  the  most  accurate  method 
that  could  be  easily  implemented  by  the  cost  analysts.  If 
"only  mission  and  performance  envelopes  are  defined  the  para¬ 
metric  approach  is  the  only  method  that  can  logically  be  used 
to  make  an  estimate”  (44:3).  Parametric  methods  are  also 
preferred  because  they  are  recommended  by  the  Cost  Analysis 
Improvement  Group  as  the  preferred  estimating  method  to  be 
used  for  Department  of  Defense  systems  during  the  development 
phase  (31:16).  "This  is  Intended  to  assure  that  major  defense 
systems  decisions  are  based  on  a  realistic  assessment  of  what 
resources  a  system  will  require;  and,  thus  whether  a  given 
system  is  cost  effective  .  .  .”(44:2). 


The  linear  regression  model  is  a  parametric  technique 
often  used  by  USAF  cost  analysts.  The  linear  regression 
■  odel  can  only  be  an  accurate  estimation  technique  if 
Independent  variables  can  be  identified  that  have  a  strong 
cauaal  linkage  to  the  dependent  variable  (flight  test  costs) 
(38).  After  the  key  independent  variables  are  Identified 
they  must  also  be  properly  specified  to  reflect  the  true 
relationship  they  have  with  the  dependent  variable  (28:132- 
140).  Major  problems  can  occur  in  linear  regression  func¬ 
tions  when  collinearity  (28:271-8,  362-400)  or  autocorrela¬ 
tion  (28:444-60)  occur.  (Collinearity  and  autocorrelation 
are  discussed  in  more  detail  in  Analytical  Techniques  in 


Chapter  II.)  Initial  tests  were  run  on  the  data  to  determine 
if  the  linear  regression  model  was  appropriate  (See 
Developing  the  Model.  Chapter  III). 

Another  parametric  technique  considered  was  the  princi¬ 
pal  component  regression  method.  This  method  is  most  appro¬ 
priately  used  when  collinearity  exists  between  the  indepen¬ 
dent  variables,  when  some  variables  need  to  be  eliminated 
because  they  contribute  very  little  information,  or  when  it 
is  necessary  to  reduce  nthe  basic  demensions  in  the  varia¬ 
bility  in  the  measured  set"  (24:235).  Two  main  advantages 
are  gained  when  the  principal  component  regression  method  is 
used:  (1)  each  component  is  free  of  collinearity  and  (2) 
"each  component  contains  a  maximum  amount  of  information 
consistent  with  being  uncorrelated  with  the  previous  ones" 
(32:255). 


A  Major  potential  problem  with  the  principal  component 
analysis  method  is  that  since  the  independent  "variables  of 
the  final  model  are  linear  combinations  of  the  original 
explanatory  variables"  the  independent  variables  in  the  final 
equation  are  not  easily  interpreted  (39:177).  Utilization  of 
the  principal  component  method  will  require  a  searching 
analysis  to  determine  the  appropriate  Interpretation  of  what 
each  component  is  measuring. 

Ridge  regression  is  a  method  used  to  correct  colline- 
arity  problems  by  "modifying  the  method  of  least  squares  to 
allow  biased  estimators  of  the  regression  coefficients" 
(28:394).  The  purpose  of  ridge  regression  is  to  Introduce  a 
small  amount  of  bias  in  exchange  for  an  increased  precision 
level  in  the  estimate  of  the  regression  coefficient.  Ridge 
regression  has  been  substantiated  as  a  beneficial  aid  to 
linear  regression  models  but  has  not  been  fully  developed  for 
use  with  principal  component  regression  models  (36). 

Chapter  III,  Developing  the  Model,  outlines  the  specific 
steps  to  be  used  in  selecting  the  appropriate  parametric 
model.  If  the  linear  regression  model  is  chosen  a  decision 
will  then  be  made  as  to  whether  ridge  regression  techniques 
are  appropriate  or  not.  If  the  principal  component  regres¬ 
sion  model  is  chosen,  the  final  equation  of  the  model  is  a 
collinear-f ree  linear  regression  model;  ridge  regression 
techniques  will  not  be  used  with  this  model.  The  sections  in 
Chepter  III  entitled  Validating  the  Model  and  Answering  the 


regression  aethod  and  the  principal  coaponent  regression 
aethod. 


Developing  the  Model 

The  purpose  of  the  model  developed  was  to  predict  long- 
range  flight  test  coats.  The  following  steps  were  used  to 
develop  the  aodel: 

1.  Identification  of  the  Independent  variables 

(cost  drivers) 

2.  Specification  of  the  variables. 

3.  Manipulation  of  the  data  to  observe  the 
following: 

a.  Outliers  in  the  data 

b.  Autocorrelation 

c.  Colllnearitj  between  independent  variables 

d.  Homoscedasticit y  of  the  data 

4.  Linear  regression  analysis 

5.  Principal  coaponent  analysis 

A  detailed  explanation  of  each  step  will  follow. 

Identification.  The  first  step  in  the  aodel  development 
was  to  select  the  appropriate  independent  variables  (cost 
drivers)  to  be  considered.  In  general,  "the  analyst  attempts 
to  identify  a  cause-and-ef f ect  relationship  between  the 
selected  Independent  and  dependent  variables"  (16:17).  To 
find  the  causal  (cause-and-ef feet)  relationships,  the  analyst 
■uat  rely  on  his  own  technical  skills  as  well  as  those  of  the 
experts  in  the  field  (16:19).  The  technical  interface 
between  the  experts  in  the  electronic  warfare  procurement  and 
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developmental  field  (engineers,  program  managers,  financial 
managers,  and  flight  test  engineers)  and  the  model  specifica¬ 
tion  was  accomplished  with  the  use  of  five  personal  inter¬ 
views.  Two  decision  rules  were  used  in  the  final  selection 
of  the  independent  variables  to  be  used  in  the  modeling 
effort  (34:32): 

Rule  #1.  A  logical  relationship  must  exist  between  the 
variable  selected  and  flight  test  costs.  In  essence,  a 
causal  relationship  must  exist  between  the  selected 
variable  and  the  flight  test  costs  that  are  to  be 
predicted  by  the  model. 

Rule  #  2.  The  selected  variable  must  also  describe  an 
electronic  warfare  equipment  characteristic  identifiable 
by  planners  in  the  early  conceptual  phase. 

With  the  use  of  the  preceding  decision  rules,  the  fol¬ 
lowing  Independent  variables  were  selected:  weight,  density, 
volume,  input  power,  type  of  equipment  (jammer,  receiver,  or 
despenser),  whether  the  system  tested  was  a  modification  of 
an  existing  system  or  a  completely  new  system,  and  whether 
the  system  was  tested  by  the  4950th  Test  Wing  (4950th  usually 
tests  projects  early  in  the  conceptual  phase). 

Specification.  The  specification  step  required  a 
functional  fora  to  be  selected  for  each  of  the  identified 
Independent  variables.  The  functional  form  of  an  independent 
variable  is  determined  by  the  behavior  of  the  dependent 
variable  aa  the  independent  variable  changes.  As  an  example, 
if  weight  is  considered  as  an  independent  variable,  then  it 
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■ust  be  discerned  (if  possible)  how  the  dependent  variable 
(flight  test  costs)  will  change  as  the  weight  of  a  system 
being  flight  tested  changes.  The  question  at  hand  is  how  are 
flight  test  costs  influenced  by  weight.  Do  flight  test  costs 
increase  as  weight  increases  or  do  the  flight  test  costs 
decrease?  Also,  in  question  is  at  what  rate  do  the  changes 
occur.  Thus,  the  question  is  whether  the  flight  test  cost 
will  increase  at  a  constant,  increasing,  or  decreasing  rate. 
In  essence,  one  is  attempting  to  find  the  first  and  second 
derivatives  of  the  function: 
cost  -  f (weight). 

A  detailed  discussion  of  the  specification  procedures  is 
beyond  the  scope  of  this  paper  but  more  detailed  explanations 
of  procedures  are  contained  in  references  (16,  17). 

One  specification  of  note  was  presented  by  Benoy  (7:3). 
He  stated,  "As  costs  increase  as  a  function  of  some  explana¬ 
tory  variable  the  rate  of  change  should  increase  at  a 
decreasing  rate."  The  rationale  was  that  this  allowed  the 
fixed  costs  associated  with  flight  tests  to  be  spread  over 
all  the  different  flight  tests  occuring  during  a  period. 

The  decision  rule  for  the  specification  of  an  indepen¬ 
dent  variable  was  as  follows: 

Decision  Rule:  The  specification  must  make  logical 
sense  and  muat  be  supported  by  expert  opinion  (expert 
opinions  were  obtained  through  personal  Interviews). 

Data  Manipulation.  The  data  was  manipulated  to  deter- 
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■ine  the  presence  of  outliers,  the  presence  of  autocorrela¬ 
tion,  the  degree  of  collinearity  present  between  the  indepen¬ 
dent  variables,  and  finally  the  degree  of  homoscedasticity  in 


the  data.  The  terms  outlier,  autocorrolation,  collinearity 
and  homoscedasticity  will  be  defined  when  used  in  the  fol¬ 
lowing  sections. 

Outliers.  Outliers  are  data  points  (observations) 
that  are  not  representative  of  the  population  being  measured 
(28:400).  "These  outlying  observations  may  involve  large 
residuals  and  often  have  dramatic  effects  on  the  fitted  least 
squares  regression  function"  (28:400).  A  means  to  evaluate 
outliers  is  to  calculate  the  "hat"  matrix  and  then  evaluate 
the  diagonal  elements  of  the  "hat"  matrix.  Kutner  (28)  gives 
a  detailed  explanation  of  both  the  theory  and  the  calcula¬ 
tions  to  obtain  the  "hat"  matrix. 

Decision  Rule:  Each  diagonal  element  of  the  "hat" 
matrix  was  compared  to  the  value  of  2p/n  where  p  is  the 
number  of  regression  parameters  and  n  is  the  number  of 
diagonal  elements  of  the  "hat"  matrix.  If  a  diagonal 
element  of  the  "hat"  matrix  is  larger  than  the  value 
calculated  for  2p/n  then  that  observation  associated 
with  the  diagonal  element  is  considered  an  outlier  and  a 
candidate  for  removal  from  the  data  set. 

Autocorrolation.  Autocorrolation  occurs  when  the 
error  terms  of  a  regression  model  are  not  random,  but  are  a 
function  of  one  or  more  of  the  independent  variables 
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(28:444).  Autocorrelation  is  common  in  time  series  data  if  a 


key  variable  has  been  omitted  (28:444).  Therefore,  the  data 
was  tested  for  autocorrolation  using  the  Durbin-Wataon  test 
for  autocorrolation.  A  detailed  explanation  of  the  test  and 
procedures  is  contained  in  reference  28  pages  450-454.  The 
value  of  D  (the  Durbin-Vatson  test  statistic)  was  calculated 
for  the  Data  Set  using  the  SPSS  software  package. 

Decision  Rule:  Use  a  Durbin-Vatson  test  table  to  accept 
or  reject  the  hypothesis  that  the  data  is  autocor- 
rolated. 


Collinearltv.  Colllnearity  exists  when  there  is  a 
linear  relationship  between  two  or  more  Independent  variables 
(39:143).  Vhen  colllnearity  exists,  one  independent  variable 
can  be  written  as  a  linear  combination  of  one  or  more  of  the 
remaining  independent  variables.  Colllnearity  causes 
increased  variation  in  the  regression  coefficients  (28:390). 
The  large  variations  in  turn  can  cause  errors  in  the  predic¬ 
tions  of  the  dependent  variable  (flight  test  cost).  There¬ 
fore,  colllnearity  is  a  problem  that  must  be  recognized  and 
taken  into  consideration  when  building  a  regression  model. 
The  following  methods  were  used  to  discern  the  presence  and 
degree  of  colllnearity  in  the  data  set: 

Test  #1.  Pairwise  colllnearity:  determined  by 
examining  the  correlation  matrix  and  the  values  of  the 
correlation  matrix. 

Decision  Rule:  Independent  variables  with  correla¬ 
tion  coefficients  greater  than  .80  indicate  excessive 
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collinearity  and  the  Independent  variable  must  be 
transformed  or  eliminated  (35:196). 


Test  #2.  Regress  each  independent  variable  in  turn 
against  all  the  remaining  Independent  variables  and  then 
compare  the  R^  (correlation  coefficient).  The  correla¬ 
tion  coefficient  (R^)  is  "the  percent  of  variance  in  the 
dependent  variable  explained  by  the  estimating  function” 
(16:33). 

Decision  Rule:  same  as  for  test  #1. 

Test  #3.  Variance  Inflation  Factor:  Similar  to  test 
#2  above. 

(VIF)k  -  (1  -  R2k)_1  k-  1,2,.. p-1 

where  R^  is  the  coefficient  or  deter¬ 
mination  when  Xk  is  regressed  on  the  p-2 
other  variables 

Decision  Rule:  Variance  Inflation  Factors  greater 
than  10  are  considered  excessive  and  the  independent 
variables  must  be  transformed  or  eliminated  (28:392). 

Homoscedasticitv.  Homoscedasticity  is  the  condi¬ 
tion  where  the  regression  error  variances  are  equal  at  all 
points  along  the  regression  line  (35:170).  The  lack  of 
homoscedasticity  is  known  as  heteroscedasticity.  The  condi¬ 
tion  of  homoscedasticity  is  necessary  for  linear  regression 
analysis  (35:170).  The  following  methods  were  used  to  test 
for  homoscedasticity  in  the  data  set: 

1.  Scatterplots:  plots  of  the  residuals  of  a 


linear  regression  against  each  of  the  independent 


variables  as  well  as  the  dependent  variable. 

Decision  Rule:  Interpretation  of  scatterplots  is 
extremely  subjective.  In  general,  each  plot  will  be 
evaluated  separately  to  deternine  if  a  visible  fan  shape 
(Figure  1)  is  present  in  the  plots.  Fan  shapes  are  an 
indication  that  heteroscedasticity  may  be  present  and 
the  data  would  need  to  be  transformed.  Possible 
transformations  are  in  the  reference  by  lutner  (28:170). 
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Figure  1.  Fan-shaped  Scatterplots. 

2.  A  more  precise  Spearman's  Correlation  Coef¬ 


ficient  Test  was  conducted  to  indicate  whether  or  not 
the  data  was  homoscedastic.  This  test  involved  rank 
ordering  each  data  point  according  to  the  appropriate 
independent  variable  (Rx)  and  also  according  to  the  size 
of  the  absolute  value  of  the  residual  of  the  dependent 
variable  (Ry)  (33:558-562).  A  correlation  coefficient, 
Rs,  was  then  calculated  as  follows: 


Rx  Ry  -  (  Rx) (  Ry)/n 
[  Rx2  -  (  Rx)2/n)[  Ry2  -  (  Ry)2/n3 
Statistical  tests  (33:560,  28:123)  can  be  used  to  test  for  a 
heteroscedastic  (non-homoscedastic)  condition  using  the  Rs 
value. 

In  the  linear  regression  model  only  those  variables 
which  could  be  measured  at  an  ordinal  level  could  be  tested 
in  this  way  for  a  heteroscedastic  condition  because  the 
Spearman's  Correlation  Coefficient  Test  requires  a  minimum  of 
ordinal  data. 

Decision  Rule:  The  hypothesis  for  the  test  is  that 
Rs  *  0;  if  |Rs|>rn,«</2  the  hypothesis  of  Rs-0  will  be 
rejected.  Accepting  the  hypothesis  will  indicate  that  a 
homoscedastic  condition  exists.  The  test  for  homosce- 
dasticity  in  this  thesis  used  values  of  n«22  (where  n 
equals  the  number  of  data  points)  and*<-.10.  Therefore 
using  the  tables  found  in  the  Meek  and  Turner  text 
(33:772)  if  a  value  of  less  .359  was  calculated  for  Rs  a 
condition  of  homoscedasticity  was  accepted. 


Multiple  Linear  Regression  Analysis 

The  fourth  step  in  developing  the  model  was  to  take  the 
remaining  data  and  perform  multiple  linear  regression  analy¬ 
sis.  The  purpose  of  multiple  linear  regression  analysis  is 
to  Identify  the  significant  relationships  between  the  inde¬ 
pendent  variables.  The  relationships  can  be  obtained  if  the 
historical  data  is  representative  of  the  population  (flight 
test  cost  producing  population)  (16:25).  The 
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assumptions  required  for  the  aultlple  regression  analysis  are 
as  follows:  (34:106) 

1.  The  variance  of  the  observations  are  equal 
(homoscedastlcity ) . 

2.  The  error  teras  are  normally  distributed. 

3.  The  expected  value  of  the  error  terms  is  zero. 

4.  Each  observation  is  independent. 

If  the  preceding  assumptions  hold,  then  the  regression 
analysis  "fits"  a  straight  line  (regression  line)  through  the 
data  points  such  that  there  is  minimum  variance  (distance 
from  the  data  point  to  the  regression  line)  in  the  sum  of 
the  squared  distances  between  the  data  points  and  any  other 
possible  straight  line.  The  process  used  to  find  the 
"fitted"  line  was  stepwise  linear  regression.  "Stepwise 
linear  regression  is  a  method  of  constructing  a  regression 
equation  which  selects  first  the  independent  variable  that 
contributes  the  most  to  explanation  of  the  variation” 
(34:37).  The  process  is  continued  until  all  the  significant 
variables  have  been  Identified  sequentially  and  brought  into 
the  regression  equation.  The  stepwise  regression  was  accom¬ 
plished  with  the  use  of  an  SPSS  software  package.  Detailed 
explanation  of  the  theory  of  multiple  linear  regression 
analysis  may  be  obtained  from  the  Kunter  (28)  reference. 


Prlncinal  Component  Analysis 


the  principal  coaponent  analysis  was  to  reaove  any  remaining 
collinearity  that  night  exist  (24:235).  The  analysis  was 
perforaed  nslng  a  BMDP4R  software  package. 

Principal  coaponent  analysis  is  based  on  the  fact  that 
"every  linear  regression  model  can  be  restated  in  terms  of  a 
set  of  orthogonal  explanatory  variables"  (39:157).  Ortho¬ 
gonality  aeans  there  are  no  linear  relationships 
(collinearity)  between  the  explanatory  variables  (39:143). 
The  regression  model  is  manipulated  in  the  following  manner 
to  insure  the  independent  variables  are  orthogonal  (not 
collinear)  (39:172-173): 

1.  Consider  the  basic  linear  regression  model 

T  m  IB  +  u  where  Y  is  an  n  x  1  vector  of 

observations  on  the 
response  variable 

x  is  an  n  x  p  matrix  with 
n  observations  and 
p  explanatory  variables 

u  is  an  n  x  1  vector  of 
the  residuals  (unex¬ 
plained  variation) 

B  is  a  p  x  1  vector  of 
regression  coefficients 

2.  Modeling  is  based  on  the  following  assumptions: 

a.  E(u)«0  expected  value  of  the  error  terms  is  zero 

b.  E(u  u’)«ff^I  error  terms  are  independent  and 

homoscedastic  (constant  variance) 

3.  There  exists  a  matrix  C  such  that 

C'(XX')C-_*_  and  CC’-C'C-I 

where  is  a  diagonal  matrix  with  "ordered 

characteristic  roots"  of  the  X'X  matrix  on  the 


diagonal. 

C  la  a  aatrlz  with  columns  that  are  normalized 
characteristic  roots  corresponding  to  the 
eigenvectors. 


4.  The  C  matrix  can  be  used  to  create  a  new  set  of 
explanatory  variables  that  are  orthogonal  as 
follows: 

(^(1),  S(2)i  ••••^(p))  “  W«XC«(X(i)i  X(2).—X(p))C 

the  W's  are  principal  components  and  finally  the 
regression  model  may  be  stated  as 

Y  m  W  «aru  where  W  -  XC 

a  •  C’B 

A  more  detailed  explanation  of  the  procedures  and 
theory  of  component  analysis  can  be  obtained  from  the  refer¬ 
ence  by  Kendall  (26). 

Validation 

Validation  Tests.  In  addition  to  the  procedures  out¬ 
lined  in  the  previous  sections  the  following  tests  will  be 
performed  to  ensure  the  model  is  valid  and  does  not  violate 
its  underlying  assumption: 

1.  Chi-Square  Test:  This  test  indicates  whether  a 
data  set  comes  from  a  specified  distribution  (33:334). 
For  the  model(s)  used  in  this  research  a  normal  distri¬ 
bution  is  required  because  the  F  and  T  statistical  tests 
(below)  are  based  on  a  normal  distribution  of  the  popu¬ 
lation  being  sampled.  The  Chi-Square  Test  will  be  used 
to  test  the  reasonableness  of  the  normality  assumption. 

2.  Kolmogorov-Smirnov  Test:  This  test  can  also  be 
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used  to  test  the  assumption  that  sample  comes  from  a 
population  with  a  normal  distribution  (33:342).  The 
Xolmogorov-Smlrnov  Test  is  more  restrictive  in  its 
assumptions  than  the  Chi-Square  Test  in  that  it  was 
continuous  rather  than  discrete  measurement  of  the  data 
(33:342).  When  possible  this  test  will  be  used  to 
test  the  reasonableness  of  the  normality  assumption. 

3.  Runs  Test:  An  underlying  assumption  of  the 
parametric  methods  and  other  statistical  tests  used  in 
this  research  is  that  a  random  sample  has  been  selected. 
"The  runs  test  provides  a  rough  check  on  the  validity  of 
this  assumption"  (33:345).  The  runs  test  will  used  to 
test  for  randomness  on  all  independent  variables  that 
can  be  dichotomized.  (Dichotomization  is  required  for 
the  runs  test  to  be  used  [33:345].) 

4.  Coefficient  of  Determination  (R2):  The  coeffi¬ 
cient  of  determination  measures  the  proportionate  reduc¬ 
tion  in  total  variation  in  a  dependent  variable 
associated  with  the  use  of  the  independent  variable(s) 
(28:97,241).  R2  will  al  ways  have  a  value  between  zero 
and  one  (inclusive).  A  value  of  zero  indicates  the 
Independent  variables  do  not  explain  any  of  the  varia¬ 
tion  in  the  dependent  variable  and  therefore  the 


be  used  to  Indicate  the  degree  of  correlation 
(colllnearity)  between  independent  variables. 

5.  F-test:  The  Fiaher  F-test  is  used  to  measure 
the  ratio  of  the  regression  mean  square  (MSR)  and  the 
error  mean  square  (MSE)  (28:88,92).  HSR  is  the  variance 
of  the  dependent  variable  that  is  explained  by  the 
independent  variable(s);  MSDE  is  the  variance  of  the 
dependent  variable  that  is  not  explained  by  the  indepen¬ 
dent  variable(s).  The  F-statistic  is  equal  to  HSR/MSE 
(28:92).  A  high  F  value  indicates  that  the  hypothesis 
that  the  independent  variable(s)  have  little  explanatory 
value  is  false  and  should  be  rejected;  a  low  F  value 
indicates  that  the  hypothesis  that  the  independent 
variable(s)  have  little  explanatory  value  is  true  and 
should  be  accepted  (28:92).  The  F-test  was  relied  on  to 
evaluate  the  utility  of  individual  independent  variables 
and  the  full  linear  regression  model. 

6.  F-test  and  R^  evaluation:  When  an  Independent 
variable  is  added  to  a  model  that  already  contains  other 
Independent  variables  the  coefficient  of  determination 
(R?)  will  either  increase  or  remain  constant;  if  R2  was 
relied  on  aa  an  accurate  indicates  of  the  total  esti¬ 
mating  power  of  a  model  the  addition  of  several  random 
Independent  variables  could  appear  (erroneously)  to 
explain  the  total  variation  of  the  dependent  variable 
(28:241).  However,  each  independent  variable  which  is 
added  will  also  Influence  the  F-ratio.  If  the  F-ratio 
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decreases  (with  the  level  of  significance  held  constant) 
tkea  the  independent  variable  that  was  added  to  the 
egaation  decreases  the  precision  of  the  estimate  by  a 
larger  extant  than  it  increases  the  explanatory  value  of 
the  modal  (28:241).  Therefore,  this  research  relied 
primarily  on  the  F-ratio  to  evaluate  the  significance  of 
independent  variables  and  used  the  R^  value  to  determine 
the  total  explained  correlation  only  if  it  (R2)  was 
achieved  at  significant  F-ratios. 

7.  T-test:  The  Students  T-test  can  be  used  to 
estimate  the  mean  in  a  normally  distributed  population 
when  only  a  sample  is  available  to  use  in  the  calcula¬ 
tions  of  the  estimate  (33:230).  The  T-test  was  used  to 
estimate  the  coefficients  of  individual  independent 
variables.  A  significant  problem  occurs  when  T-tests 
are  used  to  predict  statistical  levels  of  significance 
for  independent  variables  that  are  marked  by  collinear 
relationships  with  other  Independent  variables  (28:275- 
9).  Therefore,  T-tests  were  only  used  for  intermediate 
judgement  in  building  the  model  and  not  as  a  final 
authority  as  to  the  significance  level  of  the  indepen¬ 
dent  variables  that  were  used  to  construct  the  esti¬ 
mating  model. 

Validation  Procedures 


No  data  points  were  withheld  from  the  model  construc¬ 
tion.  Therefore,  additional  points  aust  be  collected  before 


validation  can  be  completed 


In  accordance  with  the  needs  of  the  Reconnaissance/Elec¬ 
tronic  Warfare  Program  Office  the  minimum  acceptable  criteria 
for  estiaating  accuracy  was  set  at  50Z  of  the  actual  observa¬ 
tion  (2).  The  desired  estlasting  accuracy  was  set  at  10Z  of 
the  actual  observation;  therefore,  all  confidence  Intervals 
were  constructed  with  ranges  equal  to  10Z  (  a  -20Z)  of  the 
actual  observation.  The  requirement  to  estimate  flight  test 
costs  up  to  six  years  in  advance  with  limited  information 
available  justified  the  50Z  minimum  criterion  and  made  the 
10Z  criterion  seem  extremely  difficult  to  achieve.  The 
eigenvalues,  determinants  and  significance  levels  previously 
noted  were  used  to  evaluate  the  sensitivity  of  the  model  to 
the  lmput  parameters  (independent  variables). 

Answering  the  Research  Questions 

Following  is  a  restatement  of  each  research  question  and 
an  evaluation  of  how  the  methodology  provided  for  it  to  be 

answered: 

Question  #1.  Does  the  historical  data  available  repre¬ 
sent  the  population  to  be  predicted?  The  section  titled  Data 
Collection  of  this  chapter  adequately  described  the  popula¬ 
tion  and  accounted  for  variances  between  the  sample  and  the 
population.  The  parametric  technique  using  independent 
variables  to  account  for  difference  between  the  population 
and  the  sample  was  also  discussed;  these  'dummy'  variables 
will  help  add  predictive  value  to  the  model  for  members  of 


41 


the  population  that  have  characteristics  that  are  not  posses¬ 
sed  bj  all  ■eabers  of  the  population. 

Question  #2.  Are  the  cost  estiaating  (causal)  relation¬ 
ships  predicted  bj  experts  in  the  field  consistent  with  the 
data  collected?  The  aethodology  described  in  the  section 
titled  Developing  the  Model  of  this  chapter  will  adequately 
answer  this  question.  The  statistical  tests  applied  to  the 
linear  regression  and  principal  coaponents  regression  models 
will  deteralne  if  these  predicted  causal  relationships  are 
statistically  significant. 

Question  #3.  What  type  of  aodel  is  the  most  appropriate 
for  this  problea?  The  nonparaaetr ic  techniques  were 
ellainated  in  the  section  titled  Model  Selection  of  this 
chapter.  In  the  section  titled  Developing  the  Model  in 
Chapter  III  the  procedures  were  outlined  for  selection  of  the 
aoat  appropriate  paraaetrlc  aodel. 

Question  #4.  Which  explanatory  variables  can  be  identi¬ 
fied  to  predict  flight  test  costs?  Personal  Interviews  with 
flight  test  engineers  and  the  tests  outlined  in  the  section 
titled  Developing  the  Model  of  this  chapter  were  used  to 
deteralne  and  test  the  explanatory  variables. 

Question  #5.  How  accurately  can  electronic  warfare 
flight  test  costs  be  predicted?  This  question  was  answered 
by  coapleting  the  aethodology  plan  in  the  section  titled 
Developing  the  Model  and  was  tested  for  significance  using 
the  testa  described  in  the  section  titles  Validating  the 
Modal. 


Question  #6.  Can  the  model  be  validated?  A  minimum  level 
of  validation  was  outlined  in  the  section  titled  Validating 
the  Model.  The  dynamic  nature  of  the  electronic  warfare 
equipment  will  require  that  the  data  base  be  updated  to 
ensure  it  remains  valid. 

Question  #7.  Does  the  model  fulfill  the  needs  of  the 
Reconnaissance/Electronic  Warfare  Systems  Program  Office? 
This  question  will  be  addressed  to  the  Reconnaissance/ 
Electronic  Warfare  Systems  Program  Office  when  the  model 
nears  completion.  At  this  point  the  conceptualized  model  has 
been  approved  by  expert  cost  analysts;  the  plan  to  use  the 
Aeronautical  Systems  Division  CYBER  computer  system  for  the 
model  has  also  been  approved  (2,43). 

In  summary,  the  methodology  adequately  provides  for  the 
research  questions  to  be  answered  in  a  systematic  and  logical 


manner 


IV.  Findings 


Data  Control /Out Her  Identification 

Once  the  data  (see  Appendix  A)  was  grouped  and  checked 
for  accuracy  it  was  reviewed  to  ensure  only  the  data 
appropriate  for  the  model  being  developed  was  used.  Five  of 
the  data  points  were  omitted  from  use  in  the  model  develop¬ 
ment  for  various  reasons.  These  five  data  points  are  shown 
in  the  "Outliers"  section  of  Appendix  A. 

The  first  two  data  points  (PAVE  TAC/FLIR  and  PAVE 
TAC/VATS)  were  omitted  because  they  are  both  strictly  electro 
optical  systems;  none  of  the  other  systems  in  the  study  were 
designed  primarily  as  electro-optical  systems.  After 
identifying  these  two  systems  it  became  apparent  that  the 
flight  test  procedures  were  different  enough  that  the  popula¬ 
tion  of  interest  could  not  be  broadened  to  include  these  two 
observations.  Ve  were  also  not  sure  that  the  cost  data 
collected  from  the  Reconnaissnce  and  Electronic  Warfare 
System  Program  Office  at  Wrlght-Patterson  AFB  constituted  all 
the  flight  test  costs  of  these  systems. 

The  JON  #2272  WA02  modification  to  the  ALQ  131  system 
was  considered  an  outlier  with  respect  to  the  dependent 
variable  flight  test  costs.  The  cost  of  testing  this  modi¬ 
fied  system  was  only  $15,000  which  was  not  close  to  the 
$.’51,000  spent  to  test  RR-180  Chaff  system,  the  next  least 
costly  project.  Because  of  this  disparity  (possible  Data 
recording  errors)  in  flight  test  cost  the  JON  #2272  WA02 


aodif icatlon  to  the  ALQ  131  system  was  eliminated  as  a  data 
point. 

The  two  data  points  associated  with  the  ALQ  172,  JON 
#5615  VA06  and  JON  #5615  VA07,  were  not  used  because  this 
system  represented  a  quantum  leap  in  capability  from  the 
other  systems  represented  in  the  data.  The  ALQ-172  system 
was  to  have  the  capability  to  perform  the  functions  of  a 
receiver,  jammer  and  dispenser  simultaneously.  Because  of 
this  leap  in  capability  these  two  observations  were  not 
Included  in  the  population.  In  the  data  collection  phase 
similar  systems  being  developed  for  the  F-15  and  F-16  air¬ 
craft  were  omitted  because  of  incomplete  data. 

Variable  Identification 

Independent  variables  were  identified  in  accordance  with 
the  guidance  provided  in  Chapter  III  under  Identification.  A 
Pearson's  correlation  coefficient  matrix  (Appendix  B)  was 
constructed  for  the  data  (Appendix  A)  in  order  that  the 
Independent  variables  could  be  analyzed.  If  an  independent 
variable  was  statistically  related  to  flight  test  costs  in  a 
nonsensical  manner  then  that  variable  was  not  used;  only  the 
variables  that  were  correlated  to  flight  test  costs  in  a 
logical  manner  were  used  to  develop  the  cost  estimating 
models. 

Variables  identified  as  candidates  for  either  parametric 
model  are:  (Note:  the  computer  codeword  is  shown  in  paren¬ 
theses). 


DENSITY  (DEN):  Flight  test  costs  were  expected  to  vary 
directly  with  density  because  the  more  dense  a  system  was  the 
■ore  technologically  complex  it  should  be;  this  increased 
coaplexity  would  indicate  that  flight  test  costs  would  need 
to  be  greater  to  fully  test  a  more  complex  system.  The 
correlation  between  density  and  flight  test  costs  was  .1020. 

MODIFICATION  (MOD):  Flight  test  costs  were  expected  to 
be  inversely  related  to  the  modification  indicator  variable 
because  a  modified  system  should  require  less  flight  testing 
than  a  new  system.  The  correlation  between  the  modification 
indicator  variable  and  flight  test  costs  was  -.4867. 

POWER  (POW):  Total  input  power  was  expected  to  vary 
directly  with  flight  test  costs;  if  a  system  had  greater 
input  power  it  would  be  more  likely  to  be  larger,  more 
complex  and  have  Increased  capabilities  that  would  require  a 
greater  testing  effort  and  therefore  higher  flight  test 
costs.  Power  and  flight  test  costs  were  positively  corre¬ 
lated  at  a  .2709  level. 

RECEIVER  (RECD):  The  receiver  indicator  variable  was 
thought  to  have  a  correlation  somewhat  lower  than  that  of  the 
Jammer  (JXD)  Variable.  In  general,  the  jammer  requires  more 
extensive  flight  testing  than  the  receivers.  However,  there 
was  no  expectation  as  to  whether  RECD  would  be  directly  or 
indirectly  related  to  flight  test  costs,  only  that  it  would 
have  leas  direct  correlation/influence  than  the  JXD  Variable. 
Actual  correlation  was  a  -.0618  indicating  that  very  little 
Influence  was  exerted  by  the  variable. 
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JAMMER  (JZD):  The  jammer  indicator  variable  was 

expected  to  vary  directly  with  flight  test  costs  because 
Jammers  generally  require  more  extensive  testing  than  do  the 
other  types  of  electronic  warfare  equipment.  The  jammer 
indicator  variable  was  positively  correlated  with  flight  test 
coats  at  a  .3026  level. 

4950th  FTW  PROJECT  (FTW)j  The  4950th  FTW  project  indi¬ 
cator  variable  was  expected  to  vary  directly  with  flight  test 
coats.  The  4950th  FTW  usually  tests  systems  that  are  still 
in  the  design  phase  and  have  not  been  developed  to  the  point 
that  they  will  be  ready  for  operational  use  after  the  flight 
teat  phase  (29).  Therefore  these  projects  often  require 
major  aircraft  modifications  and  test  modifications  which  are 
reflected  in  higher  test  costs.  The  correlation  between  the 
4950th  FTW  project  indicator  variable  and  flight  test  costs 
was  .6904. 

TEST  YEAR  (TESTY):  This  variable  took  on  the  value  of 
the  year  the  flight  test  was  conducted  or  the  value  of  the 
middle  year  in  which  flight  test  costs  were  conducted.  In 
recent  years  more  emphasis  has  been  placed  on  flight  testing 
of  new  systems  in  order  that  problems  can  be  corrected  before 
"they  appear  as  deficiencies  in  deployed  systems”  (12:139); 
because  of  this  increased  emphasis  in  testing  it  was  expected 
that  flight  test  costs  would  vary  directly  with  the  test 
year.  However,  the  actual  correlation  between  test  year  and 
flight  test  coats  was  -.0930.  Further  evaluation  Indicated 
that  s  possible  cause  of  the  negative  correlation  may  be  due 
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to  increased  efficiency/proficiency  In  later  testing  opera¬ 
tions  (testa  were  conducted  with  the  aid  of  "lesaons- 
learned”).  In  addition,  many  of  the  latter  tests  were  modi- 
flcations  of  systens  rather  than  new  syetena  (eodif ications 
require  less  testing  therefore  lees  flight  teat  costs). 
Despite  the  negative  correlation  of  -.0930  it  was  decided  to 
keep  the  TESTY  variable. 

TECHNICAL  YEAR  (TECHY):  System  coaplezities  were 
expected  to  increase  with  the  increased  technical  years  which 
would  require  higher  flight  test  costs.  The  positive  corre¬ 
lation  of  .4275  between  technical  year  and  flight  test  costs 
was  aa  expected. 

DENSITY  X  4950th  FTW  PROJECT  (DENFTW):  This  interaction 
tera  will  give  added  weight  (above  the  variable  DENSITY)  to 
the  denalty  of  systems  tested  by  the  4950th  FTW.  Since 
4950th  FTW  systeas  are  tested  in  the  development  stage  they 
tend  to  be  less  dense  than  the  other  systems.  Increased 
density  among  4950th  FTW  systems  was  also  expected  to  corres¬ 
pond  to  Increased  flight  test  costs.  The  actual  correlation 
was  .843. 

VOLUME  X  MODIFICATION  (VOLMOD):  This  interaction  term 
will  increase  the  weight  given  to  the  predictive  capability 
of  the  volume  of  modified  systems.  The  variable  V0L5  (below) 
was  predicted  to  be  positively  correlated  with  flight  test 
coats  because  larger  sized  equipment  would  normally  require 
more  testing.  The  consensus  was  that  VOLMOD  would  be  inverse¬ 
ly  correlated  with  costs  because  the  interaction  tern  would 
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indicate  that  for  aodified  prograas  a  larger  volume  indicated 
leaa  technological  complexity;  this  complexity  would  be 
illustrated  by  the  fact  that  more  recent  systems 
(modifications)  are  pushed  to  be  less  bulky  in  order  for  more 
efficient  use  in  flight;  this  push  for  reduced  volume  would 
indicate  a  higher  complexity  among  modified  systems.  Also 
this  expected  negative  correlation  would  tend  to  measure  the 
absence  of  the  large  volumes  of  4950th  FTW  systems,  none  of 
which  are  modifications.  The  actual  correlation  between 
V0LM0D  and  flight  test  costs  was  -.3119. 

VOLUME  X  JAMMER  (VOLJXD):  This  Interaction  term  would 
indicate  that  the  fact  that  a  system  is  a  jammer  makes  the 
importance  of  the  volume  variable  increase  as  an  indicator  of 
the  flight  test  costs.  The  impact  of  the  VOLJXD  variable  was 
expected  to  vary  directly  with  flight  test  costs  for  the  same 
reason  flight  test  costs  would  increase  with  an  increase  in 
volume;  if  a  Jammer  system  was  greater  in  volume  it  was 
likely  to  be  more  complex  and  to  require  increased  flight 
testing.  The  actual  correlation  between  VOLJXD  and  flight 
test  costs  was  .2314. 

VOLUME  X  4950th  FTW  PROJECT  (VOLFTW):  The  4950th  FTW 
systems  were  still  in  the  developmental  stage  and  not 
compacted  into  a  shape  that  would  allow  then  to  be  carried  on 
the  operational  aircraft  the  systems  were  designed  for. 
Often  C-141  and  KC-135  aircraft  were  unidentified  in  order 
that  the  system  could  be  flight  tested  (29).  This  freedom  to 
not  have  to  restrict  a  system  to  a  certain  size  (volume) 
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■eant  that  the  volume  of  a  4950th  FTW  tested  system  was 
expected  to  vary  directly  with  flight  test  costs.  The  actual 
correlation  was  .4945. 


(VOLUME)**. 5  (V0L5):  Flight  test  costs  were  expected  to 
increase  at  a  decreasing  rate  with  volume  because  a  greater 
volume  could  mean  an  increased  capability  in  a  system  that 
would  need  to  be  tested.  However,  that  increased  capability 
was  not  expected  to  increase  directly  with  volume  since  the 
fixed  cost  involved  in  testing  a  system  would  be  able  to  be 
spread  over  all  the  flight  tests  which  occurred  (7:3).  The 
actual  correlation  was  .2208. 

POWER/VOLUME  (DPWVOL):  This  term  measures  the  amount 
of  input  power  per  cubic  inch.  The  term  DPVVOL  would  be 
expected  to  be  larger  for  a  system  of  greater  capability; 
therefore,  flight  test  costs  were  expected  to  increase 
directly  with  an  increase  in  the  DPWVOL  term.  The  actual 
correlation  between  the  two  terms  was  .2052. 

(WEIGHT)**. 5  (WT5):  Flight  test  costs  were  expected  to 
increase  at  a  decreasing  rate  with  weight  because  a  greater 
weight  would  mean  increased  system  capability  that  would  need 
to  be  tested.  The  increased  capability  was  not  expected  to 
Increase  directly  with  weight  because  the  fixed  costs 
involved  in  testing  a  system  would  be  able  to  be  shared  over 
all  the  flight  tests  which  occurred  (7:3).  The  actual  corre¬ 
lation  was  .3406. 

(WEIGHT)**. 5  x  TECHNICAL  TEAR  (WT5TEC):  The  WT5TEC 
variable  was  expected  to  vary  directly  with  flight  test  costs 
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since  this  tern  would  increase  with  an  increase  in  technical 
year  or  weight.  As  technical  years  increase  the  weight  of 
electronic  warfare  systems  should  decrease  for  a  given  capa¬ 
bility  because  of  electronic  technology  and  the  need  to 
decrease  the  size  of  electronic  warfare  systems  carried  by 
aircraft.  Therefore  an  increase  in  weight  given  an  increase 
or  no  change  in  technology  would  make  this  interaction  term 
increase  and  we  would  expect  an  increase  in  flight  test  costs 
because  the  increase  in  VT5TEC  would  indicate  an  increased 
system  capability  that  must  be  flight  tested.  The  actual 
correlation  between  VT5TEC  and  flight  test  costs  was  .4374. 


Linear  Resression  Specification 


After  the  variables  were  identified  and  evaluated  for 
the  predicted  response  with  flight  test  costs  a  stepwise 
regression  was  run  using  the  Statistical  Package  for  the 
Social  Sciences  (SPSS).  The  variables  previously  identified 
were  used  in  the  SPSS  program.  The  resulting  linear  regres¬ 
sion  model  was: 

C«  -5432.48147  +  39.76962  WT5  +  360.85572  RECD  +  116657.04676 

DENFTV  +  68.13108  TECHY  -  1875.21334  FTW 

C-  flight  test  costs 

WT5-  ( weight )*♦. 5 

RECD-  receiver 


DEHFTW-  density  x  4950th  FTW  project 
TECHY-  technical  year 
FTW-  4950th  FTW  project 


The  actual  coat  and  predicted  co8t  using  the  linear 
regression  aodel  are  in  Appendix  C.  The  model  will  be 
analyzed  in  Chapter  V. 

Linear  Regression  Model  Evaluation 

A  Kolmogoro v-Smirnov  Test  was  used  to  ensure  the 
assumption  of  a  noraal  distribution  of  data  was  not  unrea¬ 
sonable  (see  Appendix  D).  This  assumption  was  required  so 
that  F  and  T  testa  could  be  applied  to  the  linear  regression 
aodel  (28:70,118).  The  D  value  for  the  test  was  .148  which 
was  less  than  d22,.05  **285;  this  indicated  that  accepting 
the  data  as  being  normally  distributed  was  not  unreasonable. 

A  runs  test  to  check  the  randomness  of  the  data  was  also 
accomplished  (see  Appendix  E).  Randomness  of  data  points  is 
also  a  requirement  of  the  linear  regression  model 
(28:31,123).  The  number  of  runs  was  eleven  (11)  which 
allowed  the  data  to  be  accepted  as  random  at  a  90Z  level  of 
confidence. 

The  R2,  and  T-Teat  data  from  the  linear  regression  model 
is  in  Appendix  F.  The  F-Test  and  Condition  Index  information 
is  in  Appendix  6.  The  value  of  .8958  with  a  corresponding 
adjusted  R^  value  of  .6633  indicates  that  the  model  would 
adequately  fill  the  requirements  of  the  Reconnaissance  and 
Electronic  Warfare  System  Program  Office,  Wr ight-Patter son 
AFB  for  a  ballpark  estimate  of  electronic  warfare  system 
flight  teat  coats  up  to  six  years  in  advance  (2). 

Aa  predicted  a  major  area  of  concern  with  the  model 
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occurred  because  of  collinearity.  Of  the  variables  selected 
FTW  and  DENFTV  were  highly  correlated  (.935).  More  moderate 
correlation  was  found  between  TECHY  and  FTW  (.572)  and 
between  TECHY  and  DENFTV  (.51 8).  The  condition  index  bounds 
of  11.420  to  136.921  indicated  that  there  was  a  very  high 
chance  of  aignificant  multi-variable  collinearity.  Variance 
Inflation  Factor  Tests  (see  Chapter  III  section  on 
Collinearity)  were  conducted  between  each  independent  vari¬ 
able  and  the  remaining  independent  variables  (see  Appendix 
H).  Vhen  FTW  (4950th  FTW  project  was  tested  against  the 
other  independent  variables  in  the  linear  regression  model)  a 
condition  index  of  11.42  was  obtained  indicating  that  some 
collinearity  was  present  in  the  model. 

Because  of  the  problems  with  collinearity  the  usefulness 
of  the  model  was  weakened.  Statistical  tests  may  be  inac¬ 
curate  and  the  correlation  coefficients  of  the  model  may  be 
related  to  the  dependent  variable  in  a  nonsensible  manner 
(28:382-92).  Because  of  this  problem  it  was  determined  that 
use  of  the  principal  components  regression  model  was 
appropriate. 

Te8ts  and  scatterplota  were  also  conducted  to  test  for 
hoaoscedaatlclty.  The  scatterplots  (see  Appendix  I)  did  not 
reveal  any  major  problems  with  heter oscedasticity . 
Spearman's  Rank  Correlation  Coefficient  Tests  (see  Chapter 
III  section  on  Homoecedastlcitv)  were  conducted  between  the 
Independent  variables  WT5,  TECHY  and  the  ranked  absolute 
values  of  the  residuals  from  the  linear  regression  model. 


The  results  (see  Appendices  J  and  K)  indicated  that  there  was 
insufficient  evidence  to  reject  the  hypothesis  that  the  data 


was  hososcedast  ic  with  respect  to  VT5  and  TECHT.  The 
Spearaan's  correlation  coefficient  calculated  for  WT5  was 
-.289;  the  Spearaan's  correlation  coefficient  calculated  for 
TECHT  was  .047.  Since  R222,.05"*359  there  18  greater  than  a 
90Z  level  of  confidence  in  the  data  being  heteroscedastic 
with  respect  to  the  independent  variables  WT5  and  TECHT.  The 
other  variables  in  the  model  could  not  be  tested  for  homosce- 
dasticity  because  they  were  either  an  indicator  variable  (FTW 
and  RECD)  or  contained  an  indicator  variable  (DENFTW). 

The  data  was  tested  for  autocorrelation  with  a  Durbin- 
Vatson  Test  (see  Chapter  III  section  on  Autocorrelation). 
The  calculated  value  of  2.02299  was  well  within  limits  at  the 
90Z  confidence  level.  As  a  result,  no  problems  with  autocor¬ 
relation  were  expected  to  exist  in  the  model. 

Principal  Component  Analysis  Model 

Because  of  the  indication  of  some  collinearity  in  the 
linear  regression  model  a  principal  components  analysis  model 
was  constructed  using  the  BMDP4R  statistical  package  on  the 
CTBER  computer.  The  BMDP4R  program  selected  only  thirteen  of 
the  sixteen  variables  for  use  in  the  model.  Using  stepwise 
regression  the  model  developed  was  : 

Flight  Test  Costs-  -429.990  -  383.980  (MOD)  -  .070  (P0W)  + 
230.145  (RECD)  +  1780.573  (FTW)  -  44.138  (TESTT)  +  46.093 
(TECHT)  +.027  (V0LM0D)  +  .005  (V0LJXD)  -  .087  (V0LFTW)  - 


1.002  (V0L5)  -  500.982  (DPWVOL) 

(cost  In  thousands  of  dollars) 
HOD  -  Systea  Modification# 

POW  -  Input  Power 
RECD  »  Receiver# 

PTW  -  4950th  Test  Wing# 

TESTY  ■  Test  Tear 

TECHY  ■  Technological  Year 

VOLMOD  m  Voluae  z  Modif ication# 

VOLTXD  a  Voluae  x  Jaaaing  System# 

VOLPTW  -  Voluae  x  4950th  Test  Wing# 

V0L5  -  (Voluae)**. 5 
DPWVOL  -  Power/Voluae 

WT5TEC  «  (Weight)**. 5  x  Technological  Year 
WT5  -  (Weight)**. 5 

#  Indicator  variables 

#  Interaction  Teras 
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The  R2  for  the  aodel  was  .8900.  A  complete  summary  of 
the  coefficients  of  the  stepwise  regression  is  shown  in 
Appendix  L. 

A  Kolaogrov  Test  was  used  to  ensure  the  assumption  of  a 
noraal  distribution  of  data  was  not  unreasonable  (see 
Appendix  H).  The  D  value  for  the  test  was  .140  which  was 
less  than  d22,.05  "  *285;  this  Indicated  that  accepting  the 
data  aa  being  noraally  distributed  was  not  unreasonable. 
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A  Rons  Test  to  check  the  randomness  of  the  data  was  also 
accomplished  (see  Appendix  N).  The  number  of  runs  was  10 
which  allowed  the  data  to  be  accepted  as  random  at  a  90Z 
level  of  confidence. 


V.  Analysis 

Research  Questions  Answered 

The  original  research  problem  was  to  develop  a  cost 
estimating  model  that  could  be  used  to  estimate  long-range 
flight  test  costs  (up  to  six  years  in  advance  of  the  actual 
testing)  for  electronic  warfare  equipment.  Seven  research 
questions  were  formulated  to  help  solve  the  aforementioned 
problem.  At  this  time,  each  of  the  original  research  ques¬ 
tions  will  be  restated  with  the  resulting  answers  acquired  as 
a  result  of  the  research  effort. 

1.  Does  the  historical  data  available  represent  the 
population  to  be  predicted? 

ANSWER:  A  total  of  27  data  points  were  available/ 
collected  (see  Appendix  A).  Of  the  27,  there  were  five 
data  points  determined  to  be  inappropriate  (see  Outlier 
Evaluation  Chapter  IV).  The  remaining  22  data  points  are 
representative  of  the  population  to  be  predicted  (2).  Each 
is  either  a  jammer,  receiver,  or  an  expendable  (dispenser) 
electronic  warfare  system  tested;  each  is  either  a  new  or 
modified  system;  and  each  data  point  includes  all  the  flight 
test  costs  associated  with  the  individual  system.  Therefore, 
the  historical  data  available  and  collected  are  represenative 
of  the  population  to  be  predicted. 

2.  Are  the  cost  estimating  (causal)  relationships 


predicted  by  the  experts  in  the  field  consistent  with  the 
data  collected? 


ANSWER:  The  data  collected  is  consistent  with  the 
causal  relationships  predicted  by  the  experts  in  the  field. 
Each  data  point  was  evaluated  in  detail  to  ensure  causal 
relationships  were  consistent  (see  Variable  Identification 
and  Outlier  Evaluation  Chapter  IV). 

3.  What  type  of  model  is  the  most  appropriate  for 
this  problem? 

ANSWER:  A  detailed  analysis  of  possible  techniques 
was  accomplished  to  determine  the  most  appropriate  technique 
for  this  research  problem  (see  Model  Selection  Chapter  III). 
The  parametric  techniques  of  linear  regression  and  principal 
component  analysis  were  determined  to  be  the  most  appropriate 
modeling  techniques.  A  detailed  explanation  of  each 
considered  technique  and  findings  are  contained  in  Chapter 
III  Model  Selection. 

4.  Which  explanatory  variables  can  be  identified  to 
predict  flight  test  costs? 

ANSWER:  Over  thirty  variables  were  identified 
Initially  as  potential  cost  drivers.  Each  of  the  variables 
selected  represented  information  that  would  likely  be  avail¬ 
able  six  years  prior  to  flight  tests.  Of  the  more  than 
thirty  variables  identified  an  initial  correlation  analysis 
was  conducted  to  determine  (1)  those  variables  which  affected 
the  flight  test  cost  in  the  manner  predicted  by  experts  in 
the  field  and  (2)  the  variables  that  had  high  colllnearity. 

Sixteen  variables  were  identified  that  affected  flight 
teat  coats  in  the  predicted  manner  (see  Variable  Identif ica- 


tlon  Chapter  IV).  The  determination  of  causality  froa  system 
characteristics  (such  as  power,  weight,  volume,  density, 
technological  complexity,  type,  modification/new  system)  to 
the  system  built  and  then  to  the  flight  test  costs  of  that 
system  was  difficult  and  is  somewhat  challengeable  because  of 
the  extrapolation  of  logic  involved. 

5.  How  accurately  can  electronic  warfare  flight  test 
costs  be  predicted  with  the  model  and  variables  selected? 

ANSWER:  Unfortunately  only  a  statistically  based 

answer  is  available  for  this  question.  The  final  absolute 
answer  will  only  be  revealed  when  the  actual  final  costs  are 
compared  with  the  model  predictions.  However,  statistically 
the  models  explain  89.58Z  of  the  flight  teBt  costs  with  the 
linear  regression  model  and  89.00Z  with  the  principal  compo¬ 
nent  analysis  model. 

6.  Can  the  model  be  validated? 

ANSWER:  Due  to  the  limited  availability  of  data,  all 
the  data  was  used  to  develop  the  models.  Therefore,  no  full 
validation  process  is  possible  at  this  time  but  should  be 
accomplished  as  soon  as  additional  data  points  become  avail¬ 
able.  There  is  no  apparent  reason  why  either  model  could  not 
be  subjected  to  a  complete  validation  procedure  at  a  later 
date. 

7.  Does  the  model  fulfill  the  needs  of  the  Recon¬ 
naissance/Electronic  Warfare  Systems  Program  Office? 

ANSWER:  Both  models  returned  results  which  are 

satisfactory  for  the  intended  use.  In  predicting  flight  test 


costs  several  years  in  advance,  a  'ballpark'  figure  Is  satis¬ 
factory  (2). 


However,  both  nodels  have  problems  associated  with  them. 
The  linear  regression  model  suffers  from  high  collinearity 
between  important  independent  variables  which  resulted  in  an 
inability  to  incorporate  several  key  variables  that  may  have 
contained  additional  explanatory  information.  The  principal 
component  analysis  model  corrected  for  the  collinearity  but 
incorporated  so  many  variables  that  the  logic  of  the 
underlying  causality  became  obscure. 

The  principal  component  analysis  model  also  is  only 
appropriate  for  use  when  point  estimates  are  desired.  Confi¬ 
dence  intervals  can  not  be  made  nor  can  'tradeoff'  analysis 
be  easily  accomplished  with  the  principal  component  model. 

Sensitivity  of  Models  To  lev  Variables 

In  this  section,  each  model  will  be  analyzed  with 
respect  to  the  incorporated  independent  variables  and  how 
each  independent  variable  responds  in  the  model  as  compared 
to  the  predictions  of  the  experts  in  the  electronic  warfare 
development  field. 

lev  Varlables-Regression  Model 
MODEL: 

Flight  Test  Costs  -  -5432.481  +  39.770  (WT5)  +  360.855  (RECD) 
+  116657.04  (DENFTW)  +  68.131  (TECHY)  +  1875.213  (FTW)  (cost 
in  thousands  of  dollars) 


WT5  -  (weight)**. 5 
RECD  -  system  was  a  receiver 
DENFTW  -  density  z  tested  by  the  4950th  FTW 
TECHY  -  technological  year 
FTW  -  tested  by  the  4950th  FTW 
All  the  Independent  variables  respond  as  predicted  by 
the  experts  (see  Variable  Selection  Chapter  IV).  Each  has  a 
direct  influence  on  the  total  flight  test  cost.  The  major 
influences  are  those  systems  tested  by  the  4950th  FTW  and 
this  is  as  predicted  since  the  4950th  FTW  tests  equipment 
that  is  in  the  early  conceptual/developmental  stages  and 
therefore  requires  much  more  extensive  testing  and  test  air¬ 
craft  modifications. 

lev  Varlables-Principal  Component  Model 
MODEL: 

Flight  Teat  Costs  -  -429.990  -  383.980  (MOD)  -  .070  (P0W)  + 
230.145  (RECD)  +  1780.573  (FTW)  -  44.173  (TESTY)  +  46.093 
(TECHY)  +  .027  (VOLMOD)  +  .005  (VOLJXD)  -  .087  (VOLFTW)  - 
1.002  (V0L5)  -  500.982  (DPWVOL)  +  7.663  ( WT5TEC)  -  481.578 
(WT5)  (cost  in  thousands  of  dollars) 

MOD  m  system  was  a  modification 
POW  -  Input  Power 
RECD  -  Receiver# 

FTW  -  tested  by  the  4950th  FTW# 

TESTY  *  year  the  flight  test  to  be  accomplished 
TECHY  -  technological  year 
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VOLMOD  -  volume  x  modification 
VOLJXD  -  volume  x  jamming  system© 

VOLFTW  ■  volume  x  tested  by  the  4950th  FTW 
V0L5  -  (volume)**. 5 
DPHVOL  -  power/volume 

WT5TEC  -  (weight)**. 5  x  technological  year 

WT5  -  (weight)**. 5 

#  Indicator  variables 
©  Interaction  terms 

The  principal  component  model  corrects  for  collinearity, 
but  the  incorporation  of  so  many  variables  makes  the  under¬ 
lying  logical  inferences  between  variables  obscure.  There¬ 
fore  it  was  extremely  important  that  only  those  variables 
causually  related  to  the  dependent  variable  be  included  in 
the  model.  With  this  in  mind,  only  those  variables  predicted 
by  experts  as  being  causually  related  to  flight  test  costs 
were  selected.  Thus,  no  absolute  method  of  determining  the 
sensitivity  exists  for  this  model  and  therefore  the  resulting 
predictions  are  point  estimates  only. 
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▼I*  Conclusions  wi  Implications 


Is  the  past,  cost  estimating  for  the  flight  test  costs 
for  electronic  warfare  systems  has  been  extremely  difficult 
due  to  the  complexity  of  the  syatems/process.  The  complexity 
Is  due  to  rapid  technological  advances  in  the  electronic 
warfare  field  and  the  requirement  to  make  estimates  of  flight 
test  cost 8  up  to  six  years  in  advance  of  the  tests  when  the 
program  is  still  in  the  conceptual  phase  (only  general  infor¬ 
mation  is  available  about  the  system  to  be  tested). 

The  two  parametric  models  developed  and  presented  should 
increase  both  the  accuracy  and  timeliness  of  future  cost 
estimates  by  the  Heconnaissance/Electronic  Warfare  Program 
Office  at  Wright-Patterson  AFB.  However,  there  are  both 
strengths  and  limitations  that  should  be  considered  when 
using  the  models. 
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Strengths 

Both  models  presented  will  give  good  point  estimates 
that  can  be  considered  'ballpark'  predictions.  As  a  system 
becomes  more  fully  conceptualized  and  developed  the  estimates 
must  be  adjusted  to  reflect  the  additional  information  avail¬ 
able.  Although  both  models  are  well  suited  for  long-range 
estimates,  great  care  must  be  exercised  if  estimates  are 
attempted  for  well  defined  or  fully  developed  systems.  The 
intent  of  both  models  is  to  ensure  estimates  are  'realistic' 
in  the  early  conceptual/developmental  periods. 


Jk*j»  hS  .%  .%  „  *  A  A  A  V.V  %  *«.  «_  \  O  O.v*,4  V*  V  V*  '«**  "*  *  "  »  »  . 


Ad  additional  strength  of  both  models  is  that  only 
general  information  about  the  system  to  be  tested  is  needed 
to  make  the  cost  estimate.  This  is  particularly  Important 
since  only  very  general  information  is  available  in  the  early 
conceptual  development. 

Next,  both  models  are  very  easy  to  use,  require  little 
time,  and  both  require  only  the  aimpllat  calculations  to 
develop  a  coat  estimate.  However,  the  estimator  must  be  wary 
of  the  ease  of  calculations  and  never  forget  the  importance 
of  good  common  sense/ judgement  when  using  the  models  to 
develop  a  cost  estimate. 

Limitations 

Both  models  give  point  estimates  that  are  'ballpark' 
figures.  Thus  the  estimates  are  by  no  means  exact  and  there¬ 
fore  the  estimates  must  be  updated  continuously  as  new/ 
revised  information  becomes  available. 

In  addition,  both  models  assume  that  past  flight  test 
costa  are  indicative  of  future  flight  test  costs.  If  any 
major  changes  occur  in  testing  procedures,  requirements,  or 
even  billing  procedures  then  great  care  must  be  used  when 
developing  estimates  with  either  model. 

The  linear  regression  model  also  suffers  from  some 
collinearity  problems.  Therefore,  not  all  of  the  available 
variables  could  be  used  in  the  model  development,  and  there 
still  exists  some  collinearity  among  the  remaining  variables 
actually  chosen.  This  problem  of  collinearity  leads  to  a 


•■all  degree  of  inaccuracy  if  any  use  other  than  point  esti- 
aatea  are  aade  vith  the  aodel.  With  respect  to  the  principal 
component  aodel,  collinearlty  has  been  corrected  for  but 
'tradeoff'  analysis  has  been  sacrificed. 


Implications  For  Management 


As  new  data  points  becoae  available,  both  models  should 
be  updated  periodically.  Periodic  updates  will  help  to 
insure  the  cost  estimates  remain  accurate.  In  addition,  a 


■onitering  systea  should  be  initiated  to  track  actual  costs 
versus  the  aodel  predictions.  Large  differences  should  be 
investigated  to  deteralne  the  underlying  cause.  As  with  any 
coat  estimating  technique,  there  is  no  substitute  for  common 
sense/ judgement.  If  the  flight  testing  requirements  change, 
then  aodlf lcations/ad justaents  must  be  made  to  the  predicted 
cost  estimates. 


Recommendations 

This  section  focuses  on  two  areas  where  changes/addi- 
tionel  work  is  required.  The  two  areas  identified  could  make 
the  process  of  cost  estimating  both  less  costly  and  more 
accurate. 

First,  standardized/requlred  data  collection/storage 
procedures  need  to  be  implemented.  Useful  cost  data  from 
past  prograas/pro jects  is  practically  nonexistent.  The  data 
that  ia  available  la  kept  by  nonstandard  means  and  is  there¬ 
fore  of  little  use.  Establishment  of  a  'lessons  learned'/ 
final-cost  report  would  be  extremely  beneficial.  Why 
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'reinvent-the-wheel*  everytime  a  new  cost  estimate  is 
required?  The  existing  data  systems  make  accurate  cost  esti¬ 
mates  generally  too  time  consuming*  inaccurate*  and  more 
costly  than  they  should  be.  Simple  guidelines/requirements 
could  pay  significant  dividends  in  both  time  and  money  saved 
in  this  area. 

Lastly*  it  is  recommended  that  the  models  developed  in 
this  document  be  expanded  as  soon  as  practical  to  include 
additional  types  of  electronic  warfare  systems.  For 
instance*  some  new  systems  are  integrated  systems  (combina¬ 
tion  of  Jammer*  receiver*  and/or  expendables).  These  types 
are  not  appropriate  for  the  present  two  models  presented  in 
this  document.  Therefore*  when  data  becomes  available  for 
the  integrated  systems  a  new  variable  to  indicate  those 
systems  should  be  incorporated  into  the  existing  models. 
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A:  Flisht 

Test  Data 
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DATAPT.  SYSTEM 

JON# 

COST 

VOLUME 

:*  .*/, '**/ 

( thousands 

(cubic 

*•  .  •  V 

**  '  .  «  V  ‘ 

of  dollars) 

inches) 

•  -  ..  .. 

1 

ALQ  117  “ 

2683VA1 1 

17625.6 

2 

ALQ  117 

2683VA12 

935 

17625.6 

3 

ALR-69(F-16) 

ASD0VA45 

1690.0 

& 

Chaff 

Wth’iiWBCStfySBtSB 

17625.6 

5 

ALR-56(F-15) 

5618WA11 

932 

4164.0 

6 

ALR-119  Lens 

921AVF21 

13824.0 

7 

PECM 

N/A(4950th)  1641 

16430.4 

8 

SABRE  CROSS 

N/A(4950th) 

598 

26264.3 

9 

Cross  Eye 

N/A(4950th) 

2907 

17266.0 

P™ 

10 

Single  Axis 

N/A(4950th) 

1317 

48570.0 

11 

ALQ  131 

2272VA03 

435 

32834.1 

12 

ALR  69 

ASD0VA46 

424 

1690.0 

13 

RR-180  Chaff 

2274VA04 

151 

1814.0 

14 

ALQ  131/CPMS 

ASD0WA29 

201 

32834.1 

••• 

15 

ALQ  131/Receirer 

2272WA03 

455 

lllll  II  — 

16 

MJB/B10  Flares 

2274EA06 

498 

1814.0 

:-*:£ 

17 

Flares  (ALE-40) 

4316EQ09 

180 

1814.0 

18 

ALQ  135 

2825VA01 

342 

10368.0 

19 

ALQ  94/137 

2827VA01 

478 

13824.0 

20 

ALR  56(Software) 

5618WA07 

487 

4164.0 

21 

ALE-45  (F-15) 

5618WA13 

961 

3456.0 

22 

ALQ-128(F-15) 

5618VA14 

192 

2765.0 

S _ 

OUTLIERS 

23 

PAVE  TAC/FLIR 

N/A 

160 

51182.0 

24 

PAVE  TAC/VATS 

2056EA06 

720 

51182.0 

t _ 

25 

ALQ  131 

2272WA02 

15 

32834.1 

iv.y.; 

26 

ALQ  172 

5615WA06 

1769 

17625.6 

:::o 

27 

ALQ  172 

5615WA07 

2995 

17625.6 

» 

v  Y-Y 
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SS*S.4 

•  *  •  V  "J 

*/*.  *'%  J 

.  •  v  .  /^-A* 

WEIGHT 

(pounds) 

DENSITY 
(lb./cu.in. ) 

MODIFICATION 
(yes(y)or  no(n) 

POWER 
(input  in 
watts) 

381.0 

.0216 

Y 

3900 

381.0 

.0216 

Y 

3900 

98.5 

.0583 

N 

895 

381.0 

.0216 

Y 

142.6 

.0342 

Y 

680 

565.0 

.0409 

Y 

10000 

mm  r  i  i\m 

.0220 

N 

8000** 

375.5 

.0143 

N 

8000** 

554.0 

.0321 

N 

8000** 

1H.1 

.0151 

N 

8000** 

573.3 

.0175 

Y 

8200 

98.5 

.0583 

N 

885 

37.0 

.0204 

N 

40 

573.3 

.0175 

Y 

8200 

573.3 

.0175 

Y 

8200 

37.0 

.0204 

Y 

40 

37.0 

.0204 

Y 

40 

387.0 

.0373 

Y 

400.0 

.0289 

Y 

7000 

142.6 

.0342 

Y 

680 

170.0 

.0492 

N 

90 

58.6 

.0212 

Y 

168 

OUTLIERS 


1277.0 

1277.0 

573.0 


381.0  .0216  N  3900 

381.0  .0216  N  3900 


DATA  PT 


TEST  YEAR 


TECH. YEAR 


gr SBmw  iSmSfi  A  *  -  J  • 

2 

1982 

1974 

3 

1982 

1980 

4 

1982 

1974 

5 

1982 

1976 

6 

1980 

1974 

7 

1981 

1981 

8 

1983 

1983 

9 

1982 

1982 

10 

1983 

1983 

11 

1982 

1971 

12 

1982 

1980 

13 

1982 

1982 

14 

1983 

1971 

15 

1982 

1971 

16 

1982 

1982 

17 

1982 

1976 

18 

1983 

1976 

19 

1983 

1972 

20 

1981 

1976 

21 

1983 

1983 

22 

1983 

1976 

OUTLIERS 


23 

1983 

1983 

24 

1981 

1981 

25 

1981 

1971 

26 

1983 

1983 

27 

1983 

1983 

*Note:  The  majority  of  this  data  was  collected  from  the 
3246th  Flight  Test  Wing,  Eglin  Air  Force  Base,  Florida. 
Other  sources  of  data  were  4950th  Flight  Test  Wing,  Wright- 
Patterson  Air  Force  Base,  Ohio. 


Appendix  B:  Pearson's  Correlation  Coef 


C 

DEN 

MOD 

c 

1.000 

0.102 

-0.487 

DEN 

0.102 

1.000 

-0.311 

MOD 

-0.487 

-0.311 

1.000 

POW 

0.271 

-0.314 

0.034 

RECD 

-0.062 

0.290 

0.179 

JXD 

0.303 

-0.213 

-0.140 

FTW 

0.690 

-0.273 

-0.624 

TESTY 

-0.093 

-0.210 

-0.066 

TECHY 

0.428 

0.221 

-0.828 

DEN FTW 

0.843 

-0.173 

-0.583 

VOLMOD 

-0.312 

-0.360 

0.620 

VOLJXD 

0.231 

-0.380 

-0.214 

VOL  FTW 

0.495 

-0.313 

-0.551 

VOL5 

0.221 

-0.558 

0.063 

DPWVOL 

0.205 

0.457 

-0.087 

WT5TEC 

0.437 

-0.273 

-0.059 

WT5 

0.341 

-C . 293 

0.070 

V0L5 

DPWVOL 

WT5TEC 

C 

0.221 

0.205 

0.437 

DEN 

-0.558 

0.457 

-0.273 

MOD 

0.063 

-0.087 

-0.059 

POW 

0.845 

0.583 

0.906 

RECD 

-0.262 

-0.080 

-0.299 

JXD 

0.628 

0.558 

0.727 

FTW 

0.455 

0.142 

0.522 

TESTY 

0.180 

-0.254 

0.063 

TECHY 

-0.307 

-0.130 

-0.194 

DEN  FTW 

0.356 

0.200 

0.487 

VOLMOD 

0.590 

0.072 

0.437 

VOLJXD 

0.761 

0.181 

0.725 

VOL  FTW 

0.537 

0.011 

0.547 

VOL5 

1.000 

0.160 

0.919 

DPWVOL 

0.160 

1.000 

0.423 

WT5TEC 

0.919 

0.423 

1.000 

T 


Y 


V 


YT*, 


Appendix  C:  Predicted  Versus  Actual  Cost 


DATA  PT. 

SYSTEM 

(JON#) 

LINEAR  REGRESSION 
PREDICTED  COST 

PRINCIPAL  COM¬ 
PONENTS  PREDIC¬ 
TED  COST 

1 

2683WA11 

746 

792 

2 

2683WA12 

746 

836 

3 

ASD0VA45 

773 

807 

4 

2683VA06 

385 

606 

5 

5618WA1 1 

581 

422 

6 

921AWF21 

555 

362 

7 

PECM 

1533 

1791 

8 

SABRE  CROSS 

786 

1479 

9 

CROSS  EYE 

2960 

2628 

10 

SINGLE  AXIS 
JAMMER 

1184 

846 

11 

2272WA03 

357 

507 

12 

ASD0VA46 

773 

807 

13 

2274WA04 

396 

567 

14 

ASD0VA29 

357 

463 

15 

2272VA03 

718 

573 

16 

2274EA06 

396 

232 

17 

431GEQ09 

-13 

-325 

tstc. 


ERROR  TERMS 


DATA  PT. 

ACTUAL  COST 

LSBF 

PRINCIPAL  COMPONENTS 

1 

1010 

+264 

+218 

2 

935 

+189 

+99 

3 

896 

+123 

+89 

4 

185 

-200 

-421 

5 

932 

+351 

+510 

6 

540 

-15 

+178 

7 

1641 

+108 

-150 

8 

598 

-188 

-881 

9 

2907 

-53 

+279 

10 

1317 

+133 

+471 

11 

435 

+78 

-72 

12 

424 

-349 

-383 

13 

151 

-245 

-416 

14 

201 

-156 

-262 

15 

455 

-263 

-118 

16 

498 

+102 

+266 

17 

180 

+193 

+505 

18 

342 

-186 

-1007 

19 

478 

+210 

-728 

20 

487 

-94 

+478 

21 

961 

+220 

-707 

22 

192 

-218 

-226 

Appendix  E:  Runs  Test  (Linear  Regression  Model) 


DATA  PT.  RANK  ORDER  ERROR  TERM 

(by  Flight  Test  (+  or  -) 

Cost) 


13 

151 

— 

17 

180 

+ 

4 

185 

— 

22 

192 

— 

14 

201 

_ 

18 

342 

— 

12 

424 

— 

11 

435 

+ 

15 

455 

— 

19 

478 

+ 

20 

487 

— 

16 

498 

+ 

6 

540 

— 

8 

598 

— 

3 

896 

+ 

5 

932 

+ 

2 

935 

+ 

21 

961 

+ 

1 

1010 

+ 

10 

1317 

+ 

7 

1641 

+ 

9 

2907 

- 

Appendix  G:  F- Test /Condition  Bounds  for 
Linear  Regression  Model 


MEAN 

STD  DEV 

LABEL 

c 

716.591 

628.407 

COST 

DEN 

0.028 

0.013 

DENSITY 

MOD 

0.636 

0.492 

MODIFICATION 

POW 

4400.364 

3734.600 

POWER 

RECD 

0.364 

0.492 

RECEIVER 

JXD 

0.490 

0.503 

JAMMER 

FTW 

0.182 

0.395 

4950th 

TESTV 

82.182 

0.795 

TEST  YEAR 

TECHY 

77.136 

4.313 

TECHNICAL  YEAR 

DENFTW 

0.004 

0.009 

VOLMOD 

9278.005 

11569.737 

VOLJXD 

9646.132 

14168.604 

VOLFTW 

4933.214 

12113.601 

V0L5 

107.787 

55.933 

DPWVOL 

0.289 

0.224 

WT5TEC 

1271.643 

516.602 

WT5 

16.612 

6.858 

Number  of  Cases  =  22 


ANALYSIS  OF  VARIANCE 


REGRESSION 

RESIDUAL 

F  =  27.51548 


OF 

SUM  OF  SQUARES 

MEAN 

SQUARE 

5 

7428831.67170 

1485766 

.33430 

16 

863959.64616 

53997 

.47789 

SIGNIF  F  =  0.0000 


CONDITION  NUMBER  BOUNDS:  11.420,  136.921 


Appendix  H:  Variance  Inflation  Factor  Tests 
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Appendix  I:  Scatterplots  for  Linear  Regression  Model 
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DATA  PT. 

WT5 

SANK 

/residual\2 
Irani  j 

/  WTsf 
Irani  ) 

RESIDUAL 

RANI-WT5 

5 

15.5 

l 

240.25 

15.5 

12 

17.5 

4 

.  306.25 

35 

1 

10 

9 

100 

30 

15 

3 

16 

9 

12 

13 

21 

25 

441 

105 

21 

14 

36 

196 

84 

22 

19 

49 

361 

133 

19 

7 

64 

49 

56 

4 

10 

81 

100 

90 

17 

21 

100 

441 

210 

2 

10 

121 

100 

110 

8 

12 

144 

144 

144 

18 

8 

169 

64 

104 

14 

3 

196 

9 

42 

10 

1 

225 

1 

15 

3 

17.5 

256 

306.25 

280 

7 

13 

289 

169 

221 

16 

21 

324 

441 

378 

20 

15.5 

361 

240.25 

294.5 

11 

3 

400 

9 

60 

9 

6 

441 

36 

126 

6 

5 

481 

25 

110 

3792 

3788 

2655 

vv. 


Appendix  L:  Principal  Component  Model  Printout 
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Appendix  M:  toleoaorov-Smirnov  Test 

(Principal  Components  Model) 


DATAPT. 

ASCENDING 
ORDER  (i) 

ORDERED 

RESIDUAL  (ei) 

i 

i-1 

n 

18 

1 

-1007 

.045 

0 

8 

2 

-881 

.091 

.045 

19 

3 

-728 

.136 

.091 

21 

4 

-707 

.182 

.136 

4 

5 

-421 

.227 

.182 

13 

6 

-416 

.273 

.227 

12 

7 

-383 

.318 

.273 

14 

8 

-262 

.367 

.318 

22 

9 

-226 

.409 

.367 

7 

10 

-150 

.455 

.409 

15 

11 

-118 

.500 

.455 

11 

12 

-72 

.545 

.500 

3 

13 

+89 

.591 

.545 

2 

14 

+99 

.636 

.591 

6 

15 

+178 

.682 

.636 

1 

16 

+218 

.727 

.682 

16 

17 

+266 

.773 

.727 

9 

18 

+279 

.818 

.773 

10 

19 

+471 

.864 

.818 

20 

20 

+478 

.909 

.864 

17 

21 

+505 

.955 

.909 

5 

22 

+510 

1.000 

.955 

Appendix  N:  Runs  Test  (Princinel  Components  Model) 


DATA  PT. 


RANK  ORDER 

(by  FLIGHT  TEST  COST) 


ERROR  TERM 
(+  or  -) 


151 

180 

185 

192 

201 

342 

424 

435 

455 

478 

487 

498 

540 

598 

896 

932 

935 

961 

1010 

1317 

1641 

2907 
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